PROXMOX VE ADMINISTRATION GUIDERELEASE 5.2May 16, 2018Proxmox Server Solutions Gmbhwww.proxmox.com Proxmox VE Administration Guide x11.5.3 Using ACLs Inside Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.5.4 Backup of Containers mount points . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.5.5 Replication of Containers mount points . . . . . . . . . . . . . . . . . . . . . . . . . 15711.6 Container Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.6.1 General Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.6.2 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15911.6.3 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16011.6.4 Mount Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16011.6.5 Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16311.6.6 Automatic Start and Shutdown of Containers . . . . . . . . . . . . . . . . . . . . . . 16411.7 Backup and Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16511.7.1 Container Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16511.7.2 Restoring Container Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16511.8 Managing Containers with pct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16611.8.1 CLI Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16711.8.2 Obtaining Debugging Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16711.9 Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16811.10Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16811.10.1File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16811.10.2Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16911.10.3Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16911.11Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17412 Proxmox VE Firewall 17512.1 Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17512.2 Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17512.2.1 Cluster Wide Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17612.2.2 Host Specific Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17712.2.3 VM/Container Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17812.3 Firewall Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17912.4 Security Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18012.5 IP Aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18112.5.1 Standard IP Alias local_network . . . . . . . . . . . . . . . . . . . . . . . . . . 18112.6 IP Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18112.6.1 Standard IP set management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18212.6.2 Standard IP set blacklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 Proxmox VE Administration Guide 87 / 3288.3.1 Volume OwnershipThere exists an ownership relation for image type volumes. Each such volume is owned by a VM orContainer. For example volume local:230/example-image.raw is owned by VM 230. Most storagebackends encodes this ownership information into the volume name.When you remove a VM or Container, the system also removes all associated volumes which are owned bythat VM or Container.8.4 Using the Command Line InterfaceIt is recommended to familiarize yourself with the concept behind storage pools and volume identifiers, but inreal life, you are not forced to do any of those low level operations on the command line. Normally, allocationand removal of volumes is done by the VM and Container management tools.Nevertheless, there is a command line tool called pvesm (“Proxmox VE Storage Manager”), which is ableto perform common storage management tasks.8.4.1 ExamplesAdd storage poolspvesm add <TYPE> <STORAGE_ID> <OPTIONS>pvesm add dir <STORAGE_ID> --path <PATH>pvesm add nfs <STORAGE_ID> --path <PATH> --server <SERVER> --export ←-<EXPORT>pvesm add lvm <STORAGE_ID> --vgname <VGNAME>pvesm add iscsi <STORAGE_ID> --portal <HOST[:PORT]> --target <TARGET ←->Disable storage poolspvesm set <STORAGE_ID> --disable 1Enable storage poolspvesm set <STORAGE_ID> --disable 0Change/set storage optionspvesm set <STORAGE_ID> <OPTIONS>pvesm set <STORAGE_ID> --shared 1pvesm set local --format qcow2pvesm set <STORAGE_ID> --content isoRemove storage pools. This does not delete any data, and does not disconnect or unmount anything. It justremoves the storage configuration.pvesm remove <STORAGE_ID>Allocate volumes Proxmox VE Administration Guide 88 / 328pvesm alloc <STORAGE_ID> <VMID> <name> <size> [--format <raw|qcow2>]Allocate a 4G volume in local storage. The name is auto-generated if you pass an empty string as <name>pvesm alloc local <VMID> ’’ 4GFree volumespvesm free <VOLUME_ID>WarningThis really destroys all volume data.List storage statuspvesm statusList storage contentspvesm list <STORAGE_ID> [--vmid <VMID>]List volumes allocated by VMIDpvesm list <STORAGE_ID> --vmid <VMID>List iso imagespvesm list <STORAGE_ID> --isoList container templatespvesm list <STORAGE_ID> --vztmplShow file system path for a volumepvesm path <VOLUME_ID>8.5 Directory BackendStorage pool type: dirProxmox VE can use local directories or locally mounted shares for storage. A directory is a file level storage,so you can store any content type like virtual disk images, containers, templates, ISO images or backup files.NoteYou can mount additional storages via standard linux /etc/fstab, and then define a directory storagefor that mount point. This way you can use any file system supported by Linux. Proxmox VE Administration Guide 89 / 328This backend assumes that the underlying directory is POSIX compatible, but nothing else. This implies thatyou cannot create snapshots at the storage level. But there exists a workaround for VM images using theqcow2 file format, because that format supports snapshots internally.TipSome storage types do not support O_DIRECT, so you can’t use cache mode none with such storages.Simply use cache mode writeback instead.We use a predefined directory layout to store different content types into different sub-directories. This layoutis used by all file level storage backends.Table 8.2: Directory layoutContent type SubdirVM images images/<VMID>/ISO images template/iso/Container templates template/cache/Backup files dump/8.5.1 ConfigurationThis backend supports all common storage properties, and adds an additional property called path tospecify the directory. This needs to be an absolute file system path.Configuration Example (/etc/pve/storage.cfg)dir: backuppath /mnt/backupcontent backupmaxfiles 7Above configuration defines a storage pool called backup. That pool can be used to store up to 7 backups(maxfiles 7) per VM. The real path for the backup files is /mnt/backup/dump/....8.5.2 File naming conventionsThis backend uses a well defined naming scheme for VM images:vm-<VMID>-<NAME>.<FORMAT><VMID>This specifies the owner VM. Proxmox VE Administration Guide 90 / 328<NAME>This can be an arbitrary name (ascii) without white space. The backend uses disk-[N] asdefault, where [N] is replaced by an integer to make the name unique.<FORMAT>Specifies the image format (raw|qcow2|vmdk).When you create a VM template, all VM images are renamed to indicate that they are now read-only, andcan be used as a base image for clones:base-<VMID>-<NAME>.<FORMAT>NoteSuch base images are used to generate cloned images. So it is important that those files are read-only,and never get modified. The backend changes the access mode to 0444, and sets the immutable flag(chattr +i) if the storage supports that.8.5.3 Storage FeaturesAs mentioned above, most file systems do not support snapshots out of the box. To workaround that problem,this backend is able to use qcow2 internal snapshot capabilities.Same applies to clones. The backend uses the qcow2 base image feature to create clones.Table 8.3: Storage features for backend dirContent types Image formats Shared Snapshots Clonesimagesrootdirvztmpl isobackupraw qcow2vmdk subvolno qcow2 qcow28.5.4 ExamplesPlease use the following command to allocate a 4GB image on storage local:# pvesm alloc local 100 vm-100-disk10.raw 4GFormatting ’/var/lib/vz/images/100/vm-100-disk10.raw’, fmt=raw size ←-=4294967296successfully created ’local:100/vm-100-disk10.raw’NoteThe image name must conform to above naming conventions. Proxmox VE Administration Guide 91 / 328The real file system path is shown with:# pvesm path local:100/vm-100-disk10.raw/var/lib/vz/images/100/vm-100-disk10.rawAnd you can remove the image with:# pvesm free local:100/vm-100-disk10.raw8.6 NFS BackendStorage pool type: nfsThe NFS backend is based on the directory backend, so it shares most properties. The directory layout andthe file naming conventions are the same. The main advantage is that you can directly configure the NFSserver properties, so the backend can mount the share automatically. There is no need to modify /etc/fstab. The backend can also test if the server is online, and provides a method to query the server forexported shares.8.6.1 ConfigurationThe backend supports all common storage properties, except the shared flag, which is always set. Addition-ally, the following properties are used to configure the NFS server:serverServer IP or DNS name. To avoid DNS lookup delays, it is usually preferable to use an IP addressinstead of a DNS name - unless you have a very reliable DNS server, or list the server in the local /etc/hosts file.exportNFS export path (as listed by pvesm nfsscan).You can also set NFS mount options:pathThe local mount point (defaults to /mnt/pve/<STORAGE_ID>/).optionsNFS mount options (see man nfs).Configuration Example (/etc/pve/storage.cfg)nfs: iso-templatespath /mnt/pve/iso-templatesserver 10.0.0.10export /space/iso-templatesoptions vers=3,softcontent iso,vztmpl Proxmox VE Administration Guide 92 / 328TipAfter an NFS request times out, NFS request are retried indefinitely by default. This can lead to unexpectedhangs on the client side. For read-only content, it is worth to consider the NFS soft option, which limitsthe number of retries to three.8.6.2 Storage FeaturesNFS does not support snapshots, but the backend uses qcow2 features to implement snapshots andcloning.Table 8.4: Storage features for backend nfsContent types Image formats Shared Snapshots Clonesimagesrootdirvztmpl isobackupraw qcow2vmdkyes qcow2 qcow28.6.3 ExamplesYou can get a list of exported NFS shares with:# pvesm nfsscan <server>8.7 CIFS BackendStorage pool type: cifsThe CIFS backend extends the directory backend, so that no manual setup of a CIFS mount is needed. Sucha storage can be added directly through the Proxmox VE API or the WebUI, with all our backend advantages,like server heartbeat check or comfortable selection of exported shares.8.7.1 ConfigurationThe backend supports all common storage properties, except the shared flag, which is always set. Addition-ally, the following CIFS special properties are available:serverServer IP or DNS name. Required.TipTo avoid DNS lookup delays, it is usually preferable to use an IP address instead of a DNS name - unlessyou have a very reliable DNS server, or list the server in the local /etc/hosts file. Proxmox VE Administration Guide 93 / 328shareCIFS share to use (get available ones with pvesm cifsscan or the WebUI). Required.usernameThe username for the CIFS storage. Optional, defaults to ‘guest’.passwordThe user password. Optional. It will be saved in a file only readable by root (/etc/pve/priv/<STORAGE_ID>.cred).domainSets the user domain (workgroup) for this storage. Optional.smbversionSMB protocol Version. Optional, default is 3. SMB1 is not supported due to security issues.pathThe local mount point. Optional, defaults to /mnt/pve/<STORAGE_ID>/.Configuration Example (/etc/pve/storage.cfg)cifs: backuppath /mnt/pve/backupserver 10.0.0.11share VMDatacontent backupusername annasmbversion 38.7.2 Storage FeaturesCIFS does not support snapshots on a storage level. But you may use qcow2 backing files if you still wantto have snapshots and cloning features available.Table 8.5: Storage features for backend cifsContent types Image formats Shared Snapshots Clonesimagesrootdirvztmpl isobackupraw qcow2vmdkyes qcow2 qcow2 Proxmox VE Administration Guide 94 / 3288.7.3 ExamplesYou can get a list of exported CIFS shares with:# pvesm cifsscan <server> [--username <username>] [--password]Then you could add this share as a storage to the whole Proxmox VE cluster with:# pvesm add cifs <storagename> --server <server> --share <share> [-- ←-username <username>] [--password]8.8 GlusterFS BackendStorage pool type: glusterfsGlusterFS is a scalable network file system. The system uses a modular design, runs on commodity hard-ware, and can provide a highly available enterprise storage at low costs. Such system is capable of scalingto several petabytes, and can handle thousands of clients.NoteAfter a node/brick crash, GlusterFS does a full rsync to make sure data is consistent. This can take avery long time with large files, so this backend is not suitable to store large VM images.8.8.1 ConfigurationThe backend supports all common storage properties, and adds the following GlusterFS specific options:serverGlusterFS volfile server IP or DNS name.server2Backup volfile server IP or DNS name.volumeGlusterFS Volume.transportGlusterFS transport: tcp, unix or rdmaConfiguration Example (/etc/pve/storage.cfg)glusterfs: Glusterserver 10.2.3.4server2 10.2.3.5volume glustervolcontent images,iso Proxmox VE Administration Guide 95 / 3288.8.2 File naming conventionsThe directory layout and the file naming conventions are inherited from the dir backend.8.8.3 Storage FeaturesThe storage provides a file level interface, but no native snapshot/clone implementation.Table 8.6: Storage features for backend glusterfsContent types Image formats Shared Snapshots Clonesimagesvztmpl isobackupraw qcow2vmdkyes qcow2 qcow28.9 Local ZFS Pool BackendStorage pool type: zfspoolThis backend allows you to access local ZFS pools (or ZFS file systems inside such pools).8.9.1 ConfigurationThe backend supports the common storage properties content, nodes, disable, and the followingZFS specific properties:poolSelect the ZFS pool/filesystem. All allocations are done within that pool.blocksizeSet ZFS blocksize parameter.sparseUse ZFS thin-provisioning. A sparse volume is a volume whose reservation is not equal to the volumesize.Configuration Example (/etc/pve/storage.cfg)zfspool: vmdatapool tank/vmdatacontent rootdir,imagessparse Proxmox VE Administration Guide 96 / 3288.9.2 File naming conventionsThe backend uses the following naming scheme for VM images:vm-<VMID>-<NAME> // normal VM imagesbase-<VMID>-<NAME> // template VM image (read-only)subvol-<VMID>-<NAME> // subvolumes (ZFS filesystem for containers)<VMID>This specifies the owner VM.<NAME>This can be an arbitrary name (ascii) without white space. The backend uses disk[N] as default,where [N] is replaced by an integer to make the name unique.8.9.3 Storage FeaturesZFS is probably the most advanced storage type regarding snapshot and cloning. The backend uses ZFSdatasets for both VM images (format raw) and container data (format subvol). ZFS properties are inher-ited from the parent dataset, so you can simply set defaults on the parent dataset.Table 8.7: Storage features for backend zfsContent types Image formats Shared Snapshots Clonesimagesrootdirraw subvol no yes yes8.9.4 ExamplesIt is recommended to create an extra ZFS file system to store your VM images:# zfs create tank/vmdataTo enable compression on that newly allocated file system:# zfs set compression=on tank/vmdataYou can get a list of available ZFS filesystems with:# pvesm zfsscan8.10 LVM BackendStorage pool type: lvm Proxmox VE Administration Guide xi12.6.3 Standard IP set ipfilter-net*. . . . . . . . . . . . . . . . . . . . . . . . . . . 18212.7 Services and Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18212.8 Tips and Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18312.8.1 How to allow FTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18312.8.2 Suricata IPS integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18312.9 Notes on IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18412.10Ports used by Proxmox VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18413 User Management 18513.1 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18513.1.1 System administrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18513.1.2 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18613.2 Authentication Realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18613.3 Two factor authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18713.4 Permission Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18713.4.1 Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18813.4.2 Privileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18813.4.3 Objects and Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18913.4.4 Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19013.4.5 What permission do I need? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19013.5 Command Line Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19113.6 Real World Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19213.6.1 Administrator Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19213.6.2 Auditors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19213.6.3 Delegate User Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19313.6.4 Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19314 High Availability 19414.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19514.2 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19614.3 Management Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19614.4 How It Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19714.4.1 Service States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19814.4.2 Local Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19914.4.3 Cluster Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20014.5 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Proxmox VE Administration Guide 97 / 328LVM is a light software layer on top of hard disks and partitions. It can be used to split available disk spaceinto smaller logical volumes. LVM is widely used on Linux and makes managing hard drives easier.Another use case is to put LVM on top of a big iSCSI LUN. That way you can easily manage space onthat iSCSI LUN, which would not be possible otherwise, because the iSCSI specification does not define amanagement interface for space allocation.8.10.1 ConfigurationThe LVM backend supports the common storage properties content, nodes, disable, and the follow-ing LVM specific properties:vgnameLVM volume group name. This must point to an existing volume group.baseBase volume. This volume is automatically activated before accessing the storage. This is mostlyuseful when the LVM volume group resides on a remote iSCSI server.saferemoveZero-out data when removing LVs. When removing a volume, this makes sure that all data gets erased.saferemove_throughputWipe throughput (cstream -t parameter value).Configuration Example (/etc/pve/storage.cfg)lvm: myspacevgname myspacecontent rootdir,images8.10.2 File naming conventionsThe backend use basically the same naming conventions as the ZFS pool backend.vm-<VMID>-<NAME> // normal VM images8.10.3 Storage FeaturesLVM is a typical block storage, but this backend does not support snapshot and clones. Unfortunately,normal LVM snapshots are quite inefficient, because they interfere all writes on the whole volume groupduring snapshot time.One big advantage is that you can use it on top of a shared storage, for example an iSCSI LUN. The backenditself implement proper cluster wide locking.TipThe newer LVM-thin backend allows snapshot and clones, but does not support shared storage. Proxmox VE Administration Guide 98 / 328Table 8.8: Storage features for backend lvmContent types Image formats Shared Snapshots Clonesimagesrootdirraw possible no no8.10.4 ExamplesList available volume groups:# pvesm lvmscan8.11 LVM thin BackendStorage pool type: lvmthinLVM normally allocates blocks when you create a volume. LVM thin pools instead allocates blocks when theyare written. This behaviour is called thin-provisioning, because volumes can be much larger than physicallyavailable space.You can use the normal LVM command line tools to manage and create LVM thin pools (see man lvmthinfor details). Assuming you already have a LVM volume group called pve, the following commands create anew LVM thin pool (size 100G) called data:lvcreate -L 100G -n data pvelvconvert --type thin-pool pve/data8.11.1 ConfigurationThe LVM thin backend supports the common storage properties content, nodes, disable, and thefollowing LVM specific properties:vgnameLVM volume group name. This must point to an existing volume group.thinpoolThe name of the LVM thin pool.Configuration Example (/etc/pve/storage.cfg)lvmthin: local-lvmthinpool datavgname pvecontent rootdir,images Proxmox VE Administration Guide 99 / 3288.11.2 File naming conventionsThe backend use basically the same naming conventions as the ZFS pool backend.vm-<VMID>-<NAME> // normal VM images8.11.3 Storage FeaturesLVM thin is a block storage, but fully supports snapshots and clones efficiently. New volumes are automati-cally initialized with zero.It must be mentioned that LVM thin pools cannot be shared across multiple nodes, so you can only use themas local storage.Table 8.9: Storage features for backend lvmthinContent types Image formats Shared Snapshots Clonesimagesrootdirraw no yes yes8.11.4 ExamplesList available LVM thin pools on volume group pve:# pvesm lvmthinscan pve8.12 Open-iSCSI initiatorStorage pool type: iscsiiSCSI is a widely employed technology used to connect to storage servers. Almost all storage vendorssupport iSCSI. There are also open source iSCSI target solutions available, e.g. OpenMediaVault, which isbased on Debian.To use this backend, you need to install the open-iscsi package. This is a standard Debian package,but it is not installed by default to save resources.# apt-get install open-iscsiLow-level iscsi management task can be done using the iscsiadm tool.8.12.1 ConfigurationThe backend supports the common storage properties content, nodes, disable, and the followingiSCSI specific properties: Proxmox VE Administration Guide 100 / 328portaliSCSI portal (IP or DNS name with optional port).targetiSCSI target.Configuration Example (/etc/pve/storage.cfg)iscsi: mynasportal 10.10.10.1target iqn.2006-01.openfiler.com:tsn.dcb5aaadddcontent noneTipIf you want to use LVM on top of iSCSI, it make sense to set content none. That way it is not possibleto create VMs using iSCSI LUNs directly.8.12.2 File naming conventionsThe iSCSI protocol does not define an interface to allocate or delete data. Instead, that needs to be done onthe target side and is vendor specific. The target simply exports them as numbered LUNs. So Proxmox VEiSCSI volume names just encodes some information about the LUN as seen by the linux kernel.8.12.3 Storage FeaturesiSCSI is a block level type storage, and provides no management interface. So it is usually best to exportone big LUN, and setup LVM on top of that LUN. You can then use the LVM plugin to manage the storage onthat iSCSI LUN.Table 8.10: Storage features for backend iscsiContent types Image formats Shared Snapshots Clonesimages none raw yes no no8.12.4 ExamplesScan a remote iSCSI portal, and returns a list of possible targets:pvesm iscsiscan -portal <HOST[:PORT]> Proxmox VE Administration Guide 101 / 3288.13 User Mode iSCSI BackendStorage pool type: iscsidirectThis backend provides basically the same functionality as the Open-iSCSI backed, but uses a user-levellibrary (package libiscsi2) to implement it.It should be noted that there are no kernel drivers involved, so this can be viewed as performance optimiza-tion. But this comes with the drawback that you cannot use LVM on top of such iSCSI LUN. So you need tomanage all space allocations at the storage server side.8.13.1 ConfigurationThe user mode iSCSI backend uses the same configuration options as the Open-iSCSI backed.Configuration Example (/etc/pve/storage.cfg)iscsidirect: faststoreportal 10.10.10.1target iqn.2006-01.openfiler.com:tsn.dcb5aaaddd8.13.2 Storage FeaturesNoteThis backend works with VMs only. Containers cannot use this driver.Table 8.11: Storage features for backend iscsidirectContent types Image formats Shared Snapshots Clonesimages raw yes no no8.14 Ceph RADOS Block Devices (RBD)Storage pool type: rbdCeph is a distributed object store and file system designed to provide excellent performance, reliability andscalability. RADOS block devices implement a feature rich block level storage, and you get the followingadvantages:• thin provisioning• resizable volumes• distributed and redundant (striped over multiple OSDs) Proxmox VE Administration Guide 102 / 328• full snapshot and clone capabilities• self healing• no single point of failure• scalable to the exabyte level• kernel and user space implementation availableNoteFor smaller deployments, it is also possible to run Ceph services directly on your Proxmox VE nodes.Recent hardware has plenty of CPU power and RAM, so running storage services and VMs on samenode is possible.8.14.1 ConfigurationThis backend supports the common storage properties nodes, disable, content, and the followingrbd specific properties:monhostList of monitor daemon IPs. Optional, only needed if Ceph is not running on the PVE cluster.poolCeph pool name.usernameRBD user Id. Optional, only needed if Ceph is not running on the PVE cluster.krbdAccess rbd through krbd kernel module. This is required if you want to use the storage for containers.Configuration Example for a external Ceph cluster (/etc/pve/storage.cfg)rbd: ceph-externalmonhost 10.1.1.20 10.1.1.21 10.1.1.22pool ceph-externalcontent imagesusername adminTipYou can use the rbd utility to do low-level management tasks. Proxmox VE Administration Guide 103 / 3288.14.2 AuthenticationIf you use cephx authentication, you need to copy the keyfile from your external Ceph cluster to a ProxmoxVE host.Create the directory /etc/pve/priv/ceph withmkdir /etc/pve/priv/cephThen copy the keyringscp <cephserver>:/etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ ←-ceph/<STORAGE_ID>.keyringThe keyring must be named to match your <STORAGE_ID>. Copying the keyring generally requires rootprivileges.If Ceph is installed locally on the PVE cluster, this is done automatically by pveceph or in the GUI.8.14.3 Storage FeaturesThe rbd backend is a block level storage, and implements full snapshot and clone functionality.Table 8.12: Storage features for backend rbdContent types Image formats Shared Snapshots Clonesimagesrootdirraw yes yes yes Proxmox VE Administration Guide 104 / 328Chapter 9Storage ReplicationThe pvesr command line tool manages the Proxmox VE storage replication framework. Storage replicationbrings redundancy for guests using local storage and reduces migration time.It replicates guest volumes to another node so that all data is available without using shared storage. Replica-tion uses snapshots to minimize traffic sent over the network. Therefore, new data is sent only incrementallyafter an initial full sync. In the case of a node failure, your guest data is still available on the replicated node.The replication will be done automatically in configurable intervals. The minimum replication interval is oneminute and the maximal interval is once a week. The format used to specify those intervals is a subset ofsystemd calendar events, see Schedule Format Section 9.2 section:Every guest can be replicated to multiple target nodes, but a guest cannot get replicated twice to the sametarget node.Each replications bandwidth can be limited, to avoid overloading a storage or server.Virtual guest with active replication cannot currently use online migration. Offline migration is supported ingeneral. If you migrate to a node where the guests data is already replicated only the changes since the lastsynchronisation (so called delta) must be sent, this reduces the required time significantly. In this casethe replication direction will also switch nodes automatically after the migration finished.For example: VM100 is currently on nodeA and gets replicated to nodeB. You migrate it to nodeB, so nowit gets automatically replicated back from nodeB to nodeA.If you migrate to a node where the guest is not replicated, the whole disk data must send over. After themigration the replication job continues to replicate this guest to the configured nodes.ImportantHigh-Availability is allowed in combination with storage replication, but it has the following implica-tions:• redistributing services after a more preferred node comes online will lead to errors.• recovery works, but there may be some data loss between the last synced time and the time anode failed.9.1 Supported Storage Types Proxmox VE Administration Guide 105 / 328Table 9.1: Storage TypesDescription PVE type Snapshots StableZFS (local) zfspool yes yes9.2 Schedule FormatProxmox VE has a very flexible replication scheduler. It is based on the systemd time calendar event format.1Calendar events may be used to refer to one or more points in time in a single expression.Such a calendar event uses the following format:[day(s)] [[start-time(s)][/repetition-time(s)]]This allows you to configure a set of days on which the job should run. You can also set one or more starttimes, it tells the replication scheduler the moments in time when a job should start. With this information wecould create a job which runs every workday at 10 PM: ’mon,tue,wed,thu,fri 22’ which could beabbreviated to: ’mon..fri 22’, most reasonable schedules can be written quite intuitive this way.NoteHours are set in 24h format.To allow easier and shorter configuration one or more repetition times can be set. They indicate that on thestart-time(s) itself and the start-time(s) plus all multiples of the repetition value replications will be done. Ifyou want to start replication at 8 AM and repeat it every 15 minutes until 9 AM you would use: ’8:00/15’Here you see also that if no hour separation (:) is used the value gets interpreted as minute. If such aseparation is used the value on the left denotes the hour(s) and the value on the right denotes the minute(s).Further, you can use*to match all possible values.To get additional ideas look at more Examples below Section 9.2.2.9.2.1 Detailed SpecificationdaysDays are specified with an abbreviated English version: sun, mon, tue, wed, thu, friand sat. You may use multiple days as a comma-separated list. A range of days can also be setby specifying the start and end day separated by “..”, for example mon..fri. Those formats can bealso mixed. If omitted ’*’ is assumed.time-formatA time format consists of hours and minutes interval lists. Hours and minutes are separated by ’:’.Both, hour and minute, can be list and ranges of values, using the same format as days. First comehours then minutes, hours can be omitted if not needed, in this case ’*’ is assumed for the value ofhours. The valid range for values is 0-23 for hours and 0-59 for minutes.1see man 7 systemd.time for more information Proxmox VE Administration Guide 106 / 3289.2.2 Examples:Table 9.2: Schedule ExamplesSchedule String Alternative Meaningmon,tue,wed,thu,fri mon..fri Every working day at 0:00sat,sun sat..sun Only on weekends at 0:00mon,wed,fri — Only on Monday, Wednesdayand Friday at 0:0012:05 12:05 Every day at 12:05 PM*/5 0/5 Every five minutesmon..wed 30/10 mon,tue,wed 30/10 Monday, Tuesday, Wednesday30, 40 and 50 minutes afterevery full hourmon..fri 8..17,22:0/15 — Every working day every 15minutes between 8 AM and 6PM and between 10 PM and 11PMfri 12..13:5/20 fri 12,13:5/20 Friday at 12:05, 12:25, 12:45,13:05, 13:25 and 13:4512,14,16,18,20,22:5 12/2:5 Every day starting at 12:05 until22:05, every 2 hours* */1 Every minute (minimum interval)9.3 Error HandlingIf a replication job encounters problems it will be placed in error state. In this state the configured replicationintervals get suspended temporarily. Then we retry the failed replication in a 30 minute interval, once thissucceeds the original schedule gets activated again.9.3.1 Possible issuesThis represents only the most common issues possible, depending on your setup there may be also anothercause.• Network is not working.• No free space left on the replication target storage.• Storage with same storage ID available on target nodeNoteYou can always use the replication log to get hints about a problems cause. Proxmox VE Administration Guide xii14.5.1 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20114.5.2 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20314.6 Fencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20514.6.1 How Proxmox VE Fences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20514.6.2 Configure Hardware Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20614.6.3 Recover Fenced Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20614.7 Start Failure Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20614.8 Error Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20714.9 Package Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20714.10Node Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20714.10.1Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20714.10.2Reboot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20814.10.3Manual Resource Movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20815 Backup and Restore 20915.1 Backup modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20915.2 Backup File Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21115.3 Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21115.3.1 Bandwidth Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21115.4 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21215.5 Hook Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21315.6 File Exclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21315.7 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21416 Important Service Daemons 21516.1 pvedaemon - Proxmox VE API Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21516.2 pveproxy - Proxmox VE API Proxy Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . 21516.2.1 Host based Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21516.2.2 SSL Cipher Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21616.2.3 Diffie-Hellman Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21616.2.4 Alternative HTTPS certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21616.3 pvestatd - Proxmox VE Status Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21616.4 spiceproxy - SPICE Proxy Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21616.4.1 Host based Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21717 Useful Command Line Tools 21817.1 pvesubscription - Subscription Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 21817.2 pveperf - Proxmox VE Benchmark Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Proxmox VE Administration Guide 107 / 3289.3.2 Migrating a guest in case of ErrorIn the case of a grave error a virtual guest may get stuck on a failed node. You then need to move it manuallyto a working node again.9.3.3 ExampleLets assume that you have two guests (VM 100 and CT 200) running on node A and replicate to node B.Node A failed and can not get back online. Now you have to migrate the guest to Node B manually.• connect to node B over ssh or open its shell via the WebUI• check if that the cluster is quorate# pvecm status• If you have no quorum we strongly advise to fix this first and make the node operable again. Only if this isnot possible at the moment you may use the following command to enforce quorum on the current node:# pvecm expected 1WarningIf expected votes are set avoid changes which affect the cluster (for example adding/removingnodes, storages, virtual guests) at all costs. Only use it to get vital guests up and running again orto resolve to quorum issue itself.• move both guest configuration files form the origin node A to node B:# mv /etc/pve/nodes/A/qemu-server/100.conf /etc/pve/nodes/B/qemu-server ←-/100.conf# mv /etc/pve/nodes/A/lxc/200.conf /etc/pve/nodes/B/lxc/200.conf• Now you can start the guests again:# qm start 100# pct start 200Remember to replace the VMIDs and node names with your respective values. Proxmox VE Administration Guide 108 / 3289.4 Managing JobsYou can use the web GUI to create, modify and remove replication jobs easily. Additionally the commandline interface (CLI) tool pvesr can be used to do this.You can find the replication panel on all levels (datacenter, node, virtual guest) in the web GUI. They differ inwhat jobs get shown: all, only node specific or only guest specific jobs.Once adding a new job you need to specify the virtual guest (if not already selected) and the target node.The replication schedule Section 9.2 can be set if the default of all 15 minutes is not desired. Youmay also impose rate limiting on a replication job, this can help to keep the storage load acceptable.A replication job is identified by an cluster-wide unique ID. This ID is composed of the VMID in addition to anjob number. This ID must only be specified manually if the CLI tool is used.9.5 Command Line Interface ExamplesCreate a replication job which will run every 5 minutes with limited bandwidth of 10 mbps (megabytes persecond) for the guest with guest ID 100.# pvesr create-local-job 100-0 pve1 --schedule "*/5" --rate 10Disable an active job with ID 100-0# pvesr disable 100-0Enable a deactivated job with ID 100-0# pvesr enable 100-0Change the schedule interval of the job with ID 100-0 to once a hour# pvesr update 100-0 --schedule ’*/00’ Proxmox VE Administration Guide 109 / 328Chapter 10Qemu/KVM Virtual MachinesQemu (short form for Quick Emulator) is an open source hypervisor that emulates a physical computer.From the perspective of the host system where Qemu is running, Qemu is a user program which has accessto a number of local resources like partitions, files, network cards which are then passed to an emulatedcomputer which sees them as if they were real devices.A guest operating system running in the emulated computer accesses these devices, and runs as it wererunning on real hardware. For instance you can pass an iso image as a parameter to Qemu, and the OSrunning in the emulated computer will see a real CDROM inserted in a CD drive.Qemu can emulate a great variety of hardware from ARM to Sparc, but Proxmox VE is only concerned with32 and 64 bits PC clone emulation, since it represents the overwhelming majority of server hardware. Theemulation of PC clones is also one of the fastest due to the availability of processor extensions which greatlyspeed up Qemu when the emulated architecture is the same as the host architecture.NoteYou may sometimes encounter the term KVM (Kernel-based Virtual Machine). It means that Qemu isrunning with the support of the virtualization processor extensions, via the Linux kvm module. In thecontext of Proxmox VE Qemu and KVM can be used interchangeably as Qemu in Proxmox VE will alwaystry to load the kvm module.Qemu inside Proxmox VE runs as a root process, since this is required to access block and PCI devices.10.1 Emulated devices and paravirtualized devicesThe PC hardware emulated by Qemu includes a mainboard, network controllers, scsi, ide and sata con-trollers, serial ports (the complete list can be seen in the kvm(1) man page) all of them emulated insoftware. All these devices are the exact software equivalent of existing hardware devices, and if the OSrunning in the guest has the proper drivers it will use the devices as if it were running on real hardware. Thisallows Qemu to runs unmodified operating systems.This however has a performance cost, as running in software what was meant to run in hardware involvesa lot of extra work for the host CPU. To mitigate this, Qemu can present to the guest operating systemparavirtualized devices, where the guest OS recognizes it is running inside Qemu and cooperates with thehypervisor. Proxmox VE Administration Guide 110 / 328Qemu relies on the virtio virtualization standard, and is thus able to present paravirtualized virtio devices,which includes a paravirtualized generic disk controller, a paravirtualized network card, a paravirtualizedserial port, a paravirtualized SCSI controller, etc . . .It is highly recommended to use the virtio devices whenever you can, as they provide a big performanceimprovement. Using the virtio generic disk controller versus an emulated IDE controller will double thesequential write throughput, as measured with bonnie++(8). Using the virtio network interface candeliver up to three times the throughput of an emulated Intel E1000 network card, as measured with iperf(1).110.2 Virtual Machines SettingsGenerally speaking Proxmox VE tries to choose sane defaults for virtual machines (VM). Make sure youunderstand the meaning of the settings you change, as it could incur a performance slowdown, or puttingyour data at risk.10.2.1 General SettingsGeneral settings of a VM include• the Node : the physical server on which the VM will run• the VM ID: a unique number in this Proxmox VE installation used to identify your VM• Name: a free form text string you can use to describe the VM• Resource Pool: a logical group of VMs1See this benchmark on the KVM wiki http://www.linux-kvm.org/page/Using_VirtIO_NIC Proxmox VE Administration Guide 111 / 32810.2.2 OS SettingsWhen creating a VM, setting the proper Operating System(OS) allows Proxmox VE to optimize some lowlevel parameters. For instance Windows OS expect the BIOS clock to use the local time, while Unix basedOS expect the BIOS clock to have the UTC time.10.2.3 Hard DiskQemu can emulate a number of storage controllers:• the IDE controller, has a design which goes back to the 1984 PC/AT disk controller. Even if this controllerhas been superseded by recent designs, each and every OS you can think of has support for it, makingit a great choice if you want to run an OS released before 2003. You can connect up to 4 devices on thiscontroller.• the SATA (Serial ATA) controller, dating from 2003, has a more modern design, allowing higher throughputand a greater number of devices to be connected. You can connect up to 6 devices on this controller.• the SCSI controller, designed in 1985, is commonly found on server grade hardware, and can connect upto 14 storage devices. Proxmox VE emulates by default a LSI 53C895A controller.A SCSI controller of type VirtIO SCSI is the recommended setting if you aim for performance and isautomatically selected for newly created Linux VMs since Proxmox VE 4.3. Linux distributions have supportfor this controller since 2012, and FreeBSD since 2014. For Windows OSes, you need to provide an extraiso containing the drivers during the installation. If you aim at maximum performance, you can select aSCSI controller of type VirtIO SCSI single which will allow you to select the IO Thread option. Whenselecting VirtIO SCSI single Qemu will create a new controller for each disk, instead of adding all disks tothe same controller.• The VirtIO Block controller, often just called VirtIO or virtio-blk, is an older type of paravirtualized controller.It has been superseded by the VirtIO SCSI Controller, in terms of features. Proxmox VE Administration Guide 112 / 328On each controller you attach a number of emulated hard disks, which are backed by a file or a block deviceresiding in the configured storage. The choice of a storage type will determine the format of the hard diskimage. Storages which present block devices (LVM, ZFS, Ceph) will require the raw disk image format,whereas files based storages (Ext4, NFS, CIFS, GlusterFS) will let you to choose either the raw disk imageformat or the QEMU image format.• the QEMU image format is a copy on write format which allows snapshots, and thin provisioning of thedisk image.• the raw disk image is a bit-to-bit image of a hard disk, similar to what you would get when executing thedd command on a block device in Linux. This format does not support thin provisioning or snapshots byitself, requiring cooperation from the storage layer for these tasks. It may, however, be up to 10% fasterthan the QEMU image format.2• the VMware image format only makes sense if you intend to import/export the disk image to other hyper-visors.Setting the Cache mode of the hard drive will impact how the host system will notify the guest systems ofblock write completions. The No cache default means that the guest system will be notified that a write iscomplete when each block reaches the physical storage write queue, ignoring the host page cache. Thisprovides a good balance between safety and speed.If you want the Proxmox VE backup manager to skip a disk when doing a backup of a VM, you can set theNo backup option on that disk.If you want the Proxmox VE storage replication mechanism to skip a disk when starting a replication job, youcan set the Skip replication option on that disk. As of Proxmox VE 5.0, replication requires the disk imagesto be on a storage of type zfspool, so adding a disk image to other storages when the VM has replicationconfigured requires to skip replication for this disk image.If your storage supports thin provisioning (see the storage chapter in the Proxmox VE guide), and your VMhas a SCSI controller you can activate the Discard option on the hard disks connected to that controller. With2See this benchmark for details http://events.linuxfoundation.org/sites/events/files/slides/-CloudOpen2013_Khoa_Huynh_v3.pdf Proxmox VE Administration Guide 113 / 328Discard enabled, when the filesystem of a VM marks blocks as unused after removing files, the emulatedSCSI controller will relay this information to the storage, which will then shrink the disk image accordingly.IO ThreadThe option IO Thread can only be used when using a disk with the VirtIO controller, or with the SCSIcontroller, when the emulated controller type is VirtIO SCSI single. With this enabled, Qemu creates oneI/O thread per storage controller, instead of a single thread for all I/O, so it increases performance whenmultiple disks are used and each disk has its own storage controller. Note that backups do not currentlywork with IO Thread enabled.10.2.4 CPUA CPU socket is a physical slot on a PC motherboard where you can plug a CPU. This CPU can then containone or many cores, which are independent processing units. Whether you have a single CPU socket with4 cores, or two CPU sockets with two cores is mostly irrelevant from a performance point of view. Howeversome software licenses depend on the number of sockets a machine has, in that case it makes sense to setthe number of sockets to what the license allows you.Increasing the number of virtual cpus (cores and sockets) will usually provide a performance improvementthough that is heavily dependent on the use of the VM. Multithreaded applications will of course benefit froma large number of virtual cpus, as for each virtual cpu you add, Qemu will create a new thread of executionon the host system. If you’re not sure about the workload of your VM, it is usually a safe bet to set the numberof Total cores to 2.NoteIt is perfectly safe if the overall number of cores of all your VMs is greater than the number of cores on theserver (e.g., 4 VMs with each 4 cores on a machine with only 8 cores). In that case the host system willbalance the Qemu execution threads between your server cores, just like if you were running a standardmultithreaded application. However, Proxmox VE will prevent you from assigning more virtual CPU coresthan physically available, as this will only bring the performance down due to the cost of context switches. Proxmox VE Administration Guide 114 / 328Resource LimitsIn addition to the number of virtual cores, you can configure how much resources a VM can get in relationto the host CPU time and also in relation to other VMs. With the cpulimit (“Host CPU Time”) option youcan limit how much CPU time the whole VM can use on the host. It is a floating point value representingCPU time in percent, so 1.0 is equal to 100%, 2.5 to 250% and so on. If a single process would fully useone single core it would have 100% CPU Time usage. If a VM with four cores utilizes all its cores fully itwould theoretically use 400%. In reality the usage may be even a bit higher as Qemu can have additionalthreads for VM peripherals besides the vCPU core ones. This setting can be useful if a VM should havemultiple vCPUs, as it runs a few processes in parallel, but the VM as a whole should not be able to run allvCPUs at 100% at the same time. Using a specific example: lets say we have a VM which would profit fromhaving 8 vCPUs, but at no time all of those 8 cores should run at full load - as this would make the server sooverloaded that other VMs and CTs would get to less CPU. So, we set the cpulimit limit to 4.0 (=400%). Ifall cores do the same heavy work they would all get 50% of a real host cores CPU time. But, if only 4 woulddo work they could still get almost 100% of a real core each.NoteVMs can, depending on their configuration, use additional threads e.g., for networking or IO operationsbut also live migration. Thus a VM can show up to use more CPU time than just its virtual CPUs coulduse. To ensure that a VM never uses more CPU time than virtual CPUs assigned set the cpulimit settingto the same value as the total core count.The second CPU resource limiting setting, cpuunits (nowadays often called CPU shares or CPU weight),controls how much CPU time a VM gets in regards to other VMs running. It is a relative weight which defaultsto 1024, if you increase this for a VM it will be prioritized by the scheduler in comparison to other VMs withlower weight. E.g., if VM 100 has set the default 1024 and VM 200 was changed to 2048, the latter VM 200would receive twice the CPU bandwidth than the first VM 100.For more information see man systemd.resource-control, here CPUQuota corresponds to cpulimit and CPUShares corresponds to our cpuunits setting, visit its Notes section for references andimplementation details.CPU TypeQemu can emulate a number different of CPU types from 486 to the latest Xeon processors. Each newprocessor generation adds new features, like hardware assisted 3d rendering, random number generation,memory protection, etc . . . Usually you should select for your VM a processor type which closely matchesthe CPU of the host system, as it means that the host CPU features (also called CPU flags ) will be availablein your VMs. If you want an exact match, you can set the CPU type to host in which case the VM will haveexactly the same CPU flags as your host system.This has a downside though. If you want to do a live migration of VMs between different hosts, your VM mightend up on a new system with a different CPU type. If the CPU flags passed to the guest are missing, theqemu process will stop. To remedy this Qemu has also its own CPU type kvm64, that Proxmox VE uses bydefaults. kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set, but is guaranteedto work everywhere.In short, if you care about live migration and moving VMs between nodes, leave the kvm64 default. If youdon’t care about live migration or have a homogeneous cluster where all nodes have the same CPU, set theCPU type to host, as in theory this will give your guests maximum performance. Proxmox VE Administration Guide 115 / 328Meltdown / Spectre related CPU flagsThere are two CPU flags related to the Meltdown and Spectre vulnerabilities3which need to be set manuallyunless the selected CPU type of your VM already enables them by default.The first, called pcid, helps to reduce the performance impact of the Meltdown mitigation called Kernel Page-Table Isolation (KPTI), which effectively hides the Kernel memory from the user space. Without PCID, KPTIis quite an expensive mechanism4.The second CPU flag is called spec-ctrl, which allows an operating system to selectively disable or restrictspeculative execution in order to limit the ability of attackers to exploit the Spectre vulnerability.There are two requirements that need to be fulfilled in order to use these two CPU flags:• The host CPU(s) must support the feature and propagate it to the guest’s virtual CPU(s)• The guest operating system must be updated to a version which mitigates the attacks and is able to utilizethe CPU featureIn order to use spec-ctrl, your CPU or system vendor also needs to provide a so-called “microcode update”5for your CPU.To check if the Proxmox VE host supports PCID, execute the following command as root:# grep ’ pcid ’ /proc/cpuinfoIf this does not return empty your host’s CPU has support for pcid.To check if the Proxmox VE host supports spec-ctrl, execute the following command as root:# grep ’ spec_ctrl ’ /proc/cpuinfoIf this does not return empty your host’s CPU has support for spec-ctrl.If you use ‘host’ or another CPU type which enables the desired flags by default, and you updated your guestOS to make use of the associated CPU features, you’re already set.Otherwise you need to set the desired CPU flag of the virtual CPU, either by editing the CPU options in theWebUI, or by setting the flags property of the cpu option in the VM configuration file.NUMAYou can also optionally emulate a NUMA6architecture in your VMs. The basics of the NUMA architecturemean that instead of having a global memory pool available to all your cores, the memory is spread intolocal banks close to each socket. This can bring speed improvements as the memory bus is not a bottleneckanymore. If your system has a NUMA architecture7we recommend to activate the option, as this will allowproper distribution of the VM resources on the host system. This option is also required to hot-plug cores orRAM in a VM.If the NUMA option is used, it is recommended to set the number of sockets to the number of sockets of thehost system.3Meltdown Attack https://meltdownattack.com/4PCID is now a critical performance/security feature on x86 https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU5You can use ‘intel-microcode’ / ‘amd-microcode’ from Debian non-free if your vendor does not provide such an update.Note that not all affected CPUs can be updated to support spec-ctrl.6https://en.wikipedia.org/wiki/Non-uniform_memory_access7if the command numactl --hardware | grep available returns more than one node, then your host systemhas a NUMA architecture Proxmox VE Administration Guide 116 / 328vCPU hot-plugModern operating systems introduced the capability to hot-plug and, to a certain extent, hot-unplug CPUsin a running systems. Virtualisation allows us to avoid a lot of the (physical) problems real hardware cancause in such scenarios. Still, this is a rather new and complicated feature, so its use should be restrictedto cases where its absolutely needed. Most of the functionality can be replicated with other, well tested andless complicated, features, see Resource Limits Section 10.2.4.In Proxmox VE the maximal number of plugged CPUs is always cores*sockets. To start a VM withless than this total core count of CPUs you may use the vpus setting, it denotes how many vCPUs shouldbe plugged in at VM start.Currently only this feature is only supported on Linux, a kernel newer than 3.10 is needed, a kernel newerthan 4.7 is recommended.You can use a udev rule as follow to automatically set new CPUs as online in the guest:SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{ ←-online}="1"Save this under /etc/udev/rules.d/ as a file ending in .rules.Note: CPU hot-remove is machine dependent and requires guest cooperation. The deletion commanddoes not guarantee CPU removal to actually happen, typically it’s a request forwarded to guest using targetdependent mechanism, e.g., ACPI on x86/amd64.10.2.5 MemoryFor each VM you have the option to set a fixed size memory or asking Proxmox VE to dynamically allocatememory based on the current RAM usage of the host.Fixed Memory Allocation Proxmox VE Administration Guide xiii18 Frequently Asked Questions 22019 Bibliography 22319.1 Books about Proxmox VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22319.2 Books about related technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22319.3 Books about related topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224A Command Line Interface 225A.1 pvesm - Proxmox VE Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225A.2 pvesubscription - Proxmox VE Subscription Manager . . . . . . . . . . . . . . . . . . . . . 234A.3 pveperf - Proxmox VE Benchmark Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235A.4 pveceph - Manage CEPH Services on Proxmox VE Nodes . . . . . . . . . . . . . . . . . . . 235A.5 pvenode - Proxmox VE Node Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 239A.6 qm - Qemu/KVM Virtual Machine Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . 241A.7 qmrestore - Restore QemuServer vzdump Backups . . . . . . . . . . . . . . . . . . . . . . 265A.8 pct - Proxmox Container Toolkit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265A.9 pveam - Proxmox VE Appliance Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280A.10 pvecm - Proxmox VE Cluster Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281A.11 pvesr - Proxmox VE Storage Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284A.12 pveum - Proxmox VE User Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288A.13 vzdump - Backup Utility for VMs and Containers . . . . . . . . . . . . . . . . . . . . . . . . 293A.14 ha-manager - Proxmox VE HA Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295B Service Daemons 299B.1 pve-firewall - Proxmox VE Firewall Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . 299B.2 pvedaemon - Proxmox VE API Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300B.3 pveproxy - Proxmox VE API Proxy Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . 301B.4 pvestatd - Proxmox VE Status Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301B.5 spiceproxy - SPICE Proxy Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302B.6 pmxcfs - Proxmox Cluster File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303B.7 pve-ha-crm - Cluster Resource Manager Daemon . . . . . . . . . . . . . . . . . . . . . . . 303B.8 pve-ha-lrm - Local Resource Manager Daemon . . . . . . . . . . . . . . . . . . . . . . . . 304C Configuration Files 305C.1 Datacenter Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305C.1.1 File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305C.1.2 Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305D Firewall Macro Definitions 308E GNU Free Documentation License 322 Proxmox VE Administration Guide 117 / 328When setting memory and minimum memory to the same amount Proxmox VE will simply allocate what youspecify to your VM.Even when using a fixed memory size, the ballooning device gets added to the VM, because it delivers usefulinformation such as how much memory the guest really uses. In general, you should leave ballooningenabled, but if you want to disable it (e.g. for debugging purposes), simply uncheck Ballooning Device orsetballoon: 0in the configuration.Automatic Memory AllocationWhen setting the minimum memory lower than memory, Proxmox VE will make sure that the minimumamount you specified is always available to the VM, and if RAM usage on the host is below 80%, willdynamically add memory to the guest up to the maximum memory specified.When the host is becoming short on RAM, the VM will then release some memory back to the host, swappingrunning processes if needed and starting the oom killer in last resort. The passing around of memorybetween host and guest is done via a special balloon kernel driver running inside the guest, which willgrab or release memory pages from the host.8When multiple VMs use the autoallocate facility, it is possible to set a Shares coefficient which indicates therelative amount of the free host memory that each VM should take. Suppose for instance you have four VMs,three of them running a HTTP server and the last one is a database server. To cache more database blocksin the database server RAM, you would like to prioritize the database VM when spare RAM is available. Forthis you assign a Shares property of 3000 to the database VM, leaving the other VMs to the Shares defaultsetting of 1000. The host server has 32GB of RAM, and is currently using 16GB, leaving 32 * 80/100 - 16 =9GB RAM to be allocated to the VMs. The database VM will get 9 * 3000 / (3000 + 1000 + 1000 + 1000) =4.5 GB extra RAM and each HTTP server will get 1/5 GB.All Linux distributions released after 2010 have the balloon kernel driver included. For Windows OSes, theballoon driver needs to be added manually and can incur a slowdown of the guest, so we don’t recommendusing it on critical systems.When allocating RAM to your VMs, a good rule of thumb is always to leave 1GB of RAM available to thehost.8A good explanation of the inner workings of the balloon driver can be found here https://rwmj.wordpress.com/2010/07/-17/virtio-balloon/ Proxmox VE Administration Guide 118 / 32810.2.6 Network DeviceEach VM can have many Network interface controllers (NIC), of four different types:• Intel E1000 is the default, and emulates an Intel Gigabit network card.• the VirtIO paravirtualized NIC should be used if you aim for maximum performance. Like all VirtIO devices,the guest OS should have the proper driver installed.• the Realtek 8139 emulates an older 100 MB/s network card, and should only be used when emulatingolder operating systems ( released before 2002 )• the vmxnet3 is another paravirtualized device, which should only be used when importing a VM fromanother hypervisor.Proxmox VE will generate for each NIC a random MAC address, so that your VM is addressable on Ethernetnetworks.The NIC you added to the VM can follow one of two different models:• in the default Bridged mode each virtual NIC is backed on the host by a tap device, ( a software loopbackdevice simulating an Ethernet NIC ). This tap device is added to a bridge, by default vmbr0 in Proxmox VE.In this mode, VMs have direct access to the Ethernet LAN on which the host is located.• in the alternative NAT mode, each virtual NIC will only communicate with the Qemu user networkingstack, where a built-in router and DHCP server can provide network access. This built-in DHCP will serveaddresses in the private 10.0.2.0/24 range. The NAT mode is much slower than the bridged mode, andshould only be used for testing. This mode is only available via CLI or the API, but not via the WebUI.You can also skip adding a network device when creating a VM by selecting No network device. Proxmox VE Administration Guide 119 / 328MultiqueueIf you are using the VirtIO driver, you can optionally activate the Multiqueue option. This option allowsthe guest OS to process networking packets using multiple virtual CPUs, providing an increase in the totalnumber of packets transferred.When using the VirtIO driver with Proxmox VE, each NIC network queue is passed to the host kernel, wherethe queue will be processed by a kernel thread spawn by the vhost driver. With this option activated, it ispossible to pass multiple network queues to the host kernel for each NIC.When using Multiqueue, it is recommended to set it to a value equal to the number of Total Cores of yourguest. You also need to set in the VM the number of multi-purpose channels on each VirtIO NIC with theethtool command:ethtool -L ens1 combined Xwhere X is the number of the number of vcpus of the VM.You should note that setting the Multiqueue parameter to a value greater than one will increase the CPUload on the host and guest systems as the traffic increases. We recommend to set this option only when theVM has to process a great number of incoming connections, such as when the VM is running as a router,reverse proxy or a busy HTTP server doing long polling.10.2.7 USB PassthroughThere are two different types of USB passthrough devices:• Host USB passthrough• SPICE USB passthroughHost USB passthrough works by giving a VM a USB device of the host. This can either be done via thevendor- and product-id, or via the host bus and port.The vendor/product-id looks like this: 0123:abcd, where 0123 is the id of the vendor, and abcd is the id ofthe product, meaning two pieces of the same usb device have the same id.The bus/port looks like this: 1-2.3.4, where 1 is the bus and 2.3.4 is the port path. This represents thephysical ports of your host (depending of the internal order of the usb controllers).If a device is present in a VM configuration when the VM starts up, but the device is not present in the host,the VM can boot without problems. As soon as the device/port is available in the host, it gets passed through.WarningUsing this kind of USB passthrough means that you cannot move a VM online to another host, sincethe hardware is only available on the host the VM is currently residing.The second type of passthrough is SPICE USB passthrough. This is useful if you use a SPICE client whichsupports it. If you add a SPICE USB port to your VM, you can passthrough a USB device from where yourSPICE client is, directly to the VM (for example an input device or hardware dongle). Proxmox VE Administration Guide 120 / 32810.2.8 BIOS and UEFIIn order to properly emulate a computer, QEMU needs to use a firmware. By default QEMU uses SeaBIOSfor this, which is an open-source, x86 BIOS implementation. SeaBIOS is a good choice for most standardsetups.There are, however, some scenarios in which a BIOS is not a good firmware to boot from, e.g. if you wantto do VGA passthrough.9In such cases, you should rather use OVMF, which is an open-source UEFIimplementation.10If you want to use OVMF, there are several things to consider:In order to save things like the boot order, there needs to be an EFI Disk. This disk will be included inbackups and snapshots, and there can only be one.You can create such a disk with the following command:qm set <vmid> -efidisk0 <storage>:1,format=<format>Where <storage> is the storage where you want to have the disk, and <format> is a format which thestorage supports. Alternatively, you can create such a disk through the web interface with Add → EFI Diskin the hardware section of a VM.When using OVMF with a virtual display (without VGA passthrough), you need to set the client resolution inthe OVMF menu(which you can reach with a press of the ESC button during boot), or you have to chooseSPICE as the display type.10.2.9 Automatic Start and Shutdown of Virtual MachinesAfter creating your VMs, you probably want them to start automatically when the host system boots. For thisyou need to select the option Start at boot from the Options Tab of your VM in the web interface, or set itwith the following command:qm set <vmid> -onboot 1Start and Shutdown OrderIn some case you want to be able to fine tune the boot order of your VMs, for instance if one of your VM isproviding firewalling or DHCP to other guest systems. For this you can use the following parameters:9Alex Williamson has a very good blog entry about this. http://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html10See the OVMF Project http://www.tianocore.org/ovmf/ Proxmox VE Administration Guide 121 / 328• Start/Shutdown order: Defines the start order priority. E.g. set it to 1 if you want the VM to be the first tobe started. (We use the reverse startup order for shutdown, so a machine with a start order of 1 would bethe last to be shut down). If multiple VMs have the same order defined on a host, they will additionally beordered by VMID in ascending order.• Startup delay: Defines the interval between this VM start and subsequent VMs starts . E.g. set it to 240if you want to wait 240 seconds before starting other VMs.• Shutdown timeout: Defines the duration in seconds Proxmox VE should wait for the VM to be offline afterissuing a shutdown command. By default this value is set to 180, which means that Proxmox VE will issuea shutdown request and wait 180 seconds for the machine to be offline. If the machine is still online afterthe timeout it will be stopped forcefully.NoteVMs managed by the HA stack do not follow the start on boot and boot order options currently. ThoseVMs will be skipped by the startup and shutdown algorithm as the HA manager itself ensures that VMsget started and stopped.Please note that machines without a Start/Shutdown order parameter will always start after those where theparameter is set. Further, this parameter can only be enforced between virtual machines running on thesame host, not cluster-wide.10.3 MigrationIf you have a cluster, you can migrate your VM to another host withqm migrate <vmid> <target>There are generally two mechanisms for this• Online Migration (aka Live Migration)• Offline Migration10.3.1 Online MigrationWhen your VM is running and it has no local resources defined (such as disks on local storage, passedthrough devices, etc.) you can initiate a live migration with the -online flag. Proxmox VE Administration Guide 122 / 328How it worksThis starts a Qemu Process on the target host with the incoming flag, which means that the process startsand waits for the memory data and device states from the source Virtual Machine (since all other resources,e.g. disks, are shared, the memory content and device state are the only things left to transmit).Once this connection is established, the source begins to send the memory content asynchronously to thetarget. If the memory on the source changes, those sections are marked dirty and there will be another passof sending data. This happens until the amount of data to send is so small that it can pause the VM on thesource, send the remaining data to the target and start the VM on the target in under a second.RequirementsFor Live Migration to work, there are some things required:• The VM has no local resources (e.g. passed through devices, local disks, etc.)• The hosts are in the same Proxmox VE cluster.• The hosts have a working (and reliable) network connection.• The target host must have the same or higher versions of the Proxmox VE packages. (It might work theother way, but this is never guaranteed)10.3.2 Offline MigrationIf you have local resources, you can still offline migrate your VMs, as long as all disk are on storages, whichare defined on both hosts. Then the migration will copy the disk over the network to the target host.10.4 Copies and ClonesVM installation is usually done using an installation media (CD-ROM) from the operation system vendor.Depending on the OS, this can be a time consuming task one might want to avoid.An easy way to deploy many VMs of the same type is to copy an existing VM. We use the term clone forsuch copies, and distinguish between linked and full clones.Full CloneThe result of such copy is an independent VM. The new VM does not share any storage resourceswith the original. Proxmox VE Administration Guide 123 / 328It is possible to select a Target Storage, so one can use this to migrate a VM to a totally differentstorage. You can also change the disk image Format if the storage driver supports several formats.NoteA full clone need to read and copy all VM image data. This is usually much slower than creating alinked clone.Some storage types allows to copy a specific Snapshot, which defaults to the current VM data. Thisalso means that the final copy never includes any additional snapshots from the original VM.Linked CloneModern storage drivers supports a way to generate fast linked clones. Such a clone is a writable copywhose initial contents are the same as the original data. Creating a linked clone is nearly instanta-neous, and initially consumes no additional space.They are called linked because the new image still refers to the original. Unmodified data blocks areread from the original image, but modification are written (and afterwards read) from a new location.This technique is called Copy-on-write.This requires that the original volume is read-only. With Proxmox VE one can convert any VM into aread-only Template). Such templates can later be used to create linked clones efficiently.NoteYou cannot delete the original template while linked clones exists.It is not possible to change the Target storage for linked clones, because this is a storage internalfeature.The Target node option allows you to create the new VM on a different node. The only restriction is that theVM is on shared storage, and that storage is also available on the target node.To avoid resource conflicts, all network interface MAC addresses gets randomized, and we generate a newUUID for the VM BIOS (smbios1) setting.10.5 Virtual Machine TemplatesOne can convert a VM into a Template. Such templates are read-only, and you can use them to create linkedclones.NoteIt is not possible to start templates, because this would modify the disk images. If you want to change thetemplate, create a linked clone and modify that. Proxmox VE Administration Guide 124 / 32810.6 Importing Virtual Machines and disk imagesA VM export from a foreign hypervisor takes usually the form of one or more disk images, with a configurationfile describing the settings of the VM (RAM, number of cores).The disk images can be in the vmdk format, if the disks come from VMware or VirtualBox, or qcow2 ifthe disks come from a KVM hypervisor. The most popular configuration format for VM exports is the OVFstandard, but in practice interoperation is limited because many settings are not implemented in the standarditself, and hypervisors export the supplementary information in non-standard extensions.Besides the problem of format, importing disk images from other hypervisors may fail if the emulated hard-ware changes too much from one hypervisor to another. Windows VMs are particularly concerned by this,as the OS is very picky about any changes of hardware. This problem may be solved by installing theMergeIDE.zip utility available from the Internet before exporting and choosing a hard disk type of IDE beforebooting the imported Windows VM.Finally there is the question of paravirtualized drivers, which improve the speed of the emulated system andare specific to the hypervisor. GNU/Linux and other free Unix OSes have all the necessary drivers installedby default and you can switch to the paravirtualized drivers right after importing the VM. For Windows VMs,you need to install the Windows paravirtualized drivers by yourself.GNU/Linux and other free Unix can usually be imported without hassle. Note that we cannot guarantee asuccessful import/export of Windows VMs in all cases due to the problems above.10.6.1 Step-by-step example of a Windows OVF importMicrosoft provides Virtual Machines downloads to get started with Windows development.We are going touse one of these to demonstrate the OVF import feature.Download the Virtual Machine zipAfter getting informed about the user agreement, choose the Windows 10 Enterprise (Evaluation - Build) forthe VMware platform, and download the zip.Extract the disk image from the zipUsing the unzip utility or any archiver of your choice, unpack the zip, and copy via ssh/scp the ovf andvmdk files to your Proxmox VE host.Import the Virtual MachineThis will create a new virtual machine, using cores, memory and VM name as read from the OVF manifest,and import the disks to the local-lvm storage. You have to configure the network manually.qm importovf 999 WinDev1709Eval.ovf local-lvmThe VM is ready to be started. Proxmox VE Administration Guide 125 / 32810.6.2 Adding an external disk image to a Virtual MachineYou can also add an existing disk image to a VM, either coming from a foreign hypervisor, or one that youcreated yourself.Suppose you created a Debian/Ubuntu disk image with the vmdebootstrap tool:vmdebootstrap --verbose \--size 10GiB --serial-console \--grub --no-extlinux \--package openssh-server \--package avahi-daemon \--package qemu-guest-agent \--hostname vm600 --enable-dhcp \--customize=./copy_pub_ssh.sh \--sparse --image vm600.rawYou can now create a new target VM for this image.qm create 600 --net0 virtio,bridge=vmbr0 --name vm600 --serial0 ←-socket \--bootdisk scsi0 --scsihw virtio-scsi-pci --ostype l26Add the disk image as unused0 to the VM, using the storage pvedir:qm importdisk 600 vm600.raw pvedirFinally attach the unused disk to the SCSI controller of the VM:qm set 600 --scsi0 pvedir:600/vm-600-disk-1.rawThe VM is ready to be started.10.7 Cloud-Init SupportCloud-Init is the defacto multi-distribution package that handles early initialization of a virtual machine in-stance. Using Cloud-Init, configuration of network devices and ssh keys on the hypervisor side is possible.When the VM starts for the first time, the Cloud-Init software inside the VM will apply those settings.Many Linux distributions provide ready-to-use Cloud-Init images, mostly designed for OpenStack. Theseimages will also work with Proxmox VE. While it may seem convenient to get such ready-to-use images, weusually recommended to prepare the images by yourself. The advantage is that you will know exactly whatyou have installed, and this helps you later to easily customize the image for your needs.Once you have created such a Cloud-Init image we recommend to convert it into a VM template. From a VMtemplate you can quickly create linked clones, so this is a fast method to roll out new VM instances. You justneed to configure the network (and maybe the ssh keys) before you start the new VM.We recommend using SSH key-based authentication to login to the VMs provisioned by Cloud-Init. It is alsopossible to set a password, but this is not as safe as using SSH key-based authentication because ProxmoxVE needs to store an encrypted version of that password inside the Cloud-Init data.Proxmox VE generates an ISO image to pass the Cloud-Init data to the VM. For that purpose all Cloud-InitVMs need to have an assigned CDROM drive. Also many Cloud-Init images assume to have a serial console,so it is recommended to add a serial console and use it as display for those VMs. Proxmox VE Administration Guide 126 / 32810.7.1 Preparing Cloud-Init TemplatesThe first step is to prepare your VM. Basically you can use any VM. Simply install the Cloud-Init packagesinside the VM that you want to prepare. On Debian/Ubuntu based systems this is as simple as:apt-get install cloud-initAlready many distributions provide ready-to-use Cloud-Init images (provided as .qcow2 files), so alterna-tively you can simply download and import such images. For the following example, we will use the cloudimage provided by Ubuntu at https://cloud-images.ubuntu.com.# download the imagewget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg- ←-amd64.img# create a new VMqm create 9000 --memory 2048 --net0 virtio,bridge=vmbr0# import the downloaded disk to local-lvm storageqm importdisk 9000 bionic-server-cloudimg-amd64.img local-lvm# finally attach the new disk to the VM as scsi driveqm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-1NoteUbuntu Cloud-Init images require the virtio-scsi-pci controller type for SCSI drives.Add Cloud-Init CDROM driveThe next step is to configure a CDROM drive which will be used to pass the Cloud-Init data to the VM.qm set 9000 --ide2 local-lvm:cloudinit Proxmox VE Administration Guide 1 / 328Chapter 1IntroductionProxmox VE is a platform to run virtual machines and containers. It is based on Debian Linux, and completelyopen source. For maximum flexibility, we implemented two virtualization technologies - Kernel-based VirtualMachine (KVM) and container-based virtualization (LXC).One main design goal was to make administration as easy as possible. You can use Proxmox VE on asingle node, or assemble a cluster of many nodes. All management tasks can be done using our web-basedmanagement interface, and even a novice user can setup and install Proxmox VE within minutes.qm pvesm pveum ha-managerpctpvecm pveceph pve-firewallUser Toolspveproxy pvedaemon pvestatd pve-ha-lrm pve-clusterLinux KernelKVMContainerAppAppVMGuest OSAppAppQemuContainerAppAppVMGuest OSAppApp Proxmox VE Administration Guide 127 / 328To be able to boot directly from the Cloud-Init image, set the bootdisk parameter to scsi0, and restrictBIOS to boot from disk only. This will speed up booting, because VM BIOS skips the testing for a bootableCDROM.qm set 9000 --boot c --bootdisk scsi0Also configure a serial console and use it as a display. Many Cloud-Init images rely on this, as it is anrequirement for OpenStack images.qm set 9000 --serial0 socket --vga serial0In a last step, it is helpful to convert the VM into a template. From this template you can then quickly createlinked clones. The deployment from VM templates is much faster than creating a full clone (copy).qm template 900010.7.2 Deploying Cloud-Init TemplatesYou can easily deploy such a template by cloning:qm clone 9000 123 --name ubuntu2Then configure the SSH public key used for authentication, and configure the IP setup:qm set 123 --sshkey ~/.ssh/id_rsa.pubqm set 123 --ipconfig0 ip=10.0.10.123/24,gw=10.0.10.1You can also configure all the Cloud-Init options using a single command only. We have simply splitted theabove example to separate the commands for reducing the line length. Also make sure to adopt the IP setupfor your specific environment. Proxmox VE Administration Guide 128 / 32810.7.3 Cloud-Init specific Optionscipassword: <string>Password to assign the user. Using this is generally not recommended. Use ssh keys instead. Alsonote that older cloud-init versions do not support hashed passwords.citype: <configdrive2 | nocloud>Specifies the cloud-init configuration format. The default depends on the configured operating systemtype (ostype. We use the nocloud format for Linux, and configdrive2 for windows.ciuser: <string>User name to change ssh keys and password for instead of the image’s configured default user.ipconfig[n]: [gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,ip=<IPv4Format/CIDR>] [,ip6=<IPv6Format/CIDR>]Specify IP addresses and gateways for the corresponding interface.IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.The special string dhcp can be used for IP addresses to use DHCP, in which case no explicit gatewayshould be provided. For IPv6 the special string auto can be used to use stateless autoconfiguration.If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using dhcpon IPv4.gw=<GatewayIPv4>Default gateway for IPv4 traffic.NoteRequires option(s): ipgw6=<GatewayIPv6>Default gateway for IPv6 traffic.NoteRequires option(s): ip6ip=<IPv4Format/CIDR> (default = dhcp)IPv4 address in CIDR format.ip6=<IPv6Format/CIDR> (default = dhcp)IPv6 address in CIDR format.nameserver: <string>Sets DNS server IP address for a container. Create will automatically use the setting from the host ifneither searchdomain nor nameserver are set. Proxmox VE Administration Guide 129 / 328searchdomain: <string>Sets DNS search domains for a container. Create will automatically use the setting from the host ifneither searchdomain nor nameserver are set.sshkeys: <string>Setup public SSH keys (one key per line, OpenSSH format).10.8 Managing Virtual Machines with qmqm is the tool to manage Qemu/Kvm virtual machines on Proxmox VE. You can create and destroy virtualmachines, and control execution (start/stop/suspend/resume). Besides that, you can use qm to set parame-ters in the associated config file. It is also possible to create and delete virtual disks.10.8.1 CLI Usage ExamplesUsing an iso file uploaded on the local storage, create a VM with a 4 GB IDE disk on the local-lvm storageqm create 300 -ide0 local-lvm:4 -net0 e1000 -cdrom local:iso/proxmox ←--mailgateway_2.1.isoStart the new VMqm start 300Send a shutdown request, then wait until the VM is stopped.qm shutdown 300 && qm wait 300Same as above, but only wait for 40 seconds.qm shutdown 300 && qm wait 300 -timeout 4010.9 ConfigurationVM configuration files are stored inside the Proxmox cluster file system, and can be accessed at /etc/pve/qemu-server/<VMID>.conf. Like other files stored inside /etc/pve/, they get automaticallyreplicated to all other cluster nodes.NoteVMIDs < 100 are reserved for internal purposes, and VMIDs need to be unique cluster wide. Proxmox VE Administration Guide 130 / 328Example VM Configurationcores: 1sockets: 1memory: 512name: webmailostype: l26bootdisk: virtio0net0: e1000=EE:D2:28:5F:B6:3E,bridge=vmbr0virtio0: local:vm-100-disk-1,size=32GThose configuration files are simple text files, and you can edit them using a normal text editor (vi, nano,. . . ). This is sometimes useful to do small corrections, but keep in mind that you need to restart the VM toapply such changes.For that reason, it is usually better to use the qm command to generate and modify those files, or do thewhole thing using the GUI. Our toolkit is smart enough to instantaneously apply most changes to runningVM. This feature is called "hot plug", and there is no need to restart the VM in that case.10.9.1 File FormatVM configuration files use a simple colon separated key/value format. Each line has the following format:# this is a commentOPTION: valueBlank lines in those files are ignored, and lines starting with a # character are treated as comments and arealso ignored.10.9.2 SnapshotsWhen you create a snapshot, qm stores the configuration at snapshot time into a separate snapshot sec-tion within the same configuration file. For example, after creating a snapshot called “testsnapshot”, yourconfiguration file will look like this:VM configuration with snapshotmemory: 512swap: 512parent: testsnaphot...[testsnaphot]memory: 512swap: 512snaptime: 1457170803...There are a few snapshot related properties like parent and snaptime. The parent property is usedto store the parent/child relationship between snapshots. snaptime is the snapshot creation time stamp(Unix epoch). Proxmox VE Administration Guide 131 / 32810.9.3 Optionsacpi: <boolean> (default = 1)Enable/disable ACPI.agent: <boolean> (default = 0)Enable/disable Qemu GuestAgent.args: <string>Arbitrary arguments passed to kvm, for example:args: -no-reboot -no-hpetNotethis option is for experts only.autostart: <boolean> (default = 0)Automatic restart after crash (currently ignored).balloon: <integer> (0 - N)Amount of target RAM for the VM in MB. Using zero disables the ballon driver.bios: <ovmf | seabios> (default = seabios)Select BIOS implementation.boot: [acdn]{1,4} (default = cdn)Boot on floppy (a), hard disk (c), CD-ROM (d), or network (n).bootdisk: (ide|sata|scsi|virtio)\d+Enable booting from specified disk.cdrom: <volume>This is an alias for option -ide2cipassword: <string>cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keysinstead. Also note that older cloud-init versions do not support hashed passwords.citype: <configdrive2 | nocloud>Specifies the cloud-init configuration format. The default depends on the configured operating systemtype (ostype. We use the nocloud format for Linux, and configdrive2 for windows.ciuser: <string>cloud-init: User name to change ssh keys and password for instead of the image’s configured defaultuser. Proxmox VE Administration Guide 132 / 328cores: <integer> (1 - N) (default = 1)The number of cores per socket.cpu: [cputype=]<enum> [,flags=<+FLAG[;-FLAG...]>] [,hidden=<1|0>]Emulated CPU type.cputype=<486 | Broadwell | Broadwell-IBRS | Broadwell-noTSX |Broadwell-noTSX-IBRS | Conroe | EPYC | EPYC-IBPB | Haswell |Haswell-IBRS | Haswell-noTSX | Haswell-noTSX-IBRS | IvyBridge |IvyBridge-IBRS | Nehalem | Nehalem-IBRS | Opteron_G1 |Opteron_G2 | Opteron_G3 | Opteron_G4 | Opteron_G5 | Penryn |SandyBridge | SandyBridge-IBRS | Skylake-Client | Skylake-Client-IBRS | Skylake-Server | Skylake-Server-IBRS | Westmere |Westmere-IBRS | athlon | core2duo | coreduo | host | kvm32 |kvm64 | max | pentium | pentium2 | pentium3 | phenom | qemu32 |qemu64> (default = kvm64)Emulated CPU type.flags=<+FLAG[;-FLAG...]>List of additional CPU flags separated by ;. Use +FLAG to enable, -FLAG to disable a flag.Currently supported flags: pcid, spec-ctrl.hidden=<boolean> (default = 0)Do not identify as a KVM virtual machine.cpulimit: <number> (0 - 128) (default = 0)Limit of CPU usage.NoteIf the computer has 2 CPUs, it has total of 2 CPU time. Value 0 indicates no CPU limit.cpuunits: <integer> (2 - 262144) (default = 1024)CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, themore CPU time this VM gets. Number is relative to weights of all the other running VMs.description: <string>Description for the VM. Only used on the configuration web interface. This is saved as comment insidethe configuration file.efidisk0: [file=]<volume> [,format=<enum>] [,size=<DiskSize>]Configure a Disk for storing EFI varsfile=<volume>The drive’s backing volume.format=<cloop | cow | qcow | qcow2 | qed | raw | vmdk>The drive’s backing file’s data format. Proxmox VE Administration Guide 133 / 328size=<DiskSize>Disk size. This is purely informational and has no effect.freeze: <boolean>Freeze CPU at startup (use c monitor command to start execution).hostpci[n]: [host=]<HOSTPCIID[;HOSTPCIID2...]> [,pcie=<1|0>][,rombar=<1|0>] [,romfile=<string>] [,x-vga=<1|0>]Map host PCI devices into guest.NoteThis option allows direct access to host hardware. So it is no longer possible to migrate suchmachines - use with special care.CautionExperimental! User reported problems with this option.host=<HOSTPCIID[;HOSTPCIID2...]>Host PCI device pass through. The PCI ID of a host’s PCI device or a list of PCI virtual functionsof the host. HOSTPCIID syntax is:bus:dev.func (hexadecimal numbers)You can us the lspci command to list existing PCI devices.pcie=<boolean> (default = 0)Choose the PCI-express bus (needs the q35 machine model).rombar=<boolean> (default = 1)Specify whether or not the device’s ROM will be visible in the guest’s memory map.romfile=<string>Custom pci device rom filename (must be located in /usr/share/kvm/).x-vga=<boolean> (default = 0)Enable vfio-vga device support.hotplug: <string> (default = network,disk,usb)Selectively enable hotplug features. This is a comma separated list of hotplug features: network,disk, cpu, memory and usb. Use 0 to disable hotplug completely. Value 1 is an alias for the defaultnetwork,disk,usb.hugepages: <1024 | 2 | any>Enable/disable hugepages memory. Proxmox VE Administration Guide 134 / 328ide[n]: [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,mbps=<mbps>] [,mbps_max=<mbps>] [,mbps_rd=<mbps>][,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>] [,mbps_wr_max=<mbps>][,media=<cdrom|disk>] [,model=<model>] [,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as IDE hard disk or CD-ROM (n is 0 to 3).aio=<native | threads>AIO type to use.backup=<boolean>Whether the drive should be included when making backups.bps=<bps>Maximum r/w speed in bytes per second.bps_max_length=<seconds>Maximum length of I/O bursts in seconds.bps_rd=<bps>Maximum read speed in bytes per second.bps_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.bps_wr=<bps>Maximum write speed in bytes per second.bps_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.cache=<directsync | none | unsafe | writeback | writethrough>The drive’s cache modecyls=<integer>Force the drive’s physical geometry to have a specific cylinder count.detect_zeroes=<boolean>Controls whether to detect and try to optimize writes of zeroes.discard=<ignore | on>Controls whether to pass discard/trim requests to the underlying storage. Proxmox VE Administration Guide 135 / 328file=<volume>The drive’s backing volume.format=<cloop | cow | qcow | qcow2 | qed | raw | vmdk>The drive’s backing file’s data format.heads=<integer>Force the drive’s physical geometry to have a specific head count.iops=<iops>Maximum r/w I/O in operations per second.iops_max=<iops>Maximum unthrottled r/w I/O pool in operations per second.iops_max_length=<seconds>Maximum length of I/O bursts in seconds.iops_rd=<iops>Maximum read I/O in operations per second.iops_rd_max=<iops>Maximum unthrottled read I/O pool in operations per second.iops_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.iops_wr=<iops>Maximum write I/O in operations per second.iops_wr_max=<iops>Maximum unthrottled write I/O pool in operations per second.iops_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.mbps=<mbps>Maximum r/w speed in megabytes per second.mbps_max=<mbps>Maximum unthrottled r/w pool in megabytes per second.mbps_rd=<mbps>Maximum read speed in megabytes per second.mbps_rd_max=<mbps>Maximum unthrottled read pool in megabytes per second.mbps_wr=<mbps>Maximum write speed in megabytes per second.mbps_wr_max=<mbps>Maximum unthrottled write pool in megabytes per second. Proxmox VE Administration Guide 136 / 328media=<cdrom | disk> (default = disk)The drive’s media type.model=<model>The drive’s reported model name, url-encoded, up to 40 bytes long.replicate=<boolean> (default = 1)Whether the drive should considered for replication jobs.rerror=<ignore | report | stop>Read error action.secs=<integer>Force the drive’s physical geometry to have a specific sector count.serial=<serial>The drive’s reported serial number, url-encoded, up to 20 bytes long.shared=<boolean> (default = 0)Mark this locally-managed volume as available on all nodes.WarningThis option does not share the volume automatically, it assumes it is shared already!size=<DiskSize>Disk size. This is purely informational and has no effect.snapshot=<boolean>Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporaryand will be discarded when the VM is shutdown.trans=<auto | lba | none>Force disk geometry bios translation mode.werror=<enospc | ignore | report | stop>Write error action.ipconfig[n]: [gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,ip=<IPv4Format/CIDR>] [,ip6=<IPv6Format/CIDR>]cloud-init: Specify IP addresses and gateways for the corresponding interface.IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.The special string dhcp can be used for IP addresses to use DHCP, in which case no explicit gatewayshould be provided. For IPv6 the special string auto can be used to use stateless autoconfiguration.If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using dhcpon IPv4.gw=<GatewayIPv4>Default gateway for IPv4 traffic. Proxmox VE Administration Guide 2 / 3281.1 Central ManagementWhile many people start with a single node, Proxmox VE can scale out to a large set of clustered nodes.The cluster stack is fully integrated and ships with the default installation.Unique Multi-Master DesignThe integrated web-based management interface gives you a clean overview of all your KVM guestsand Linux containers and even of your whole cluster. You can easily manage your VMs and con-tainers, storage or cluster from the GUI. There is no need to install a separate, complex, and priceymanagement server.Proxmox Cluster File System (pmxcfs)Proxmox VE uses the unique Proxmox Cluster file system (pmxcfs), a database-driven file system forstoring configuration files. This enables you to store the configuration of thousands of virtual machines.By using corosync, these files are replicated in real time on all cluster nodes. The file system storesall data inside a persistent database on disk, nonetheless, a copy of the data resides in RAM whichprovides a maximum storage size is 30MB - more than enough for thousands of VMs.Proxmox VE is the only virtualization platform using this unique cluster file system.Web-based Management InterfaceProxmox VE is simple to use. Management tasks can be done via the included web based manage-ment interface - there is no need to install a separate management tool or any additional managementnode with huge databases. The multi-master tool allows you to manage your whole cluster from anynode of your cluster. The central web-based management - based on the JavaScript Framework (Ex-tJS) - empowers you to control all functionalities from the GUI and overview history and syslogs of eachsingle node. This includes running backup or restore jobs, live-migration or HA triggered activities.Command LineFor advanced users who are used to the comfort of the Unix shell or Windows Powershell, ProxmoxVE provides a command line interface to manage all the components of your virtual environment. Thiscommand line interface has intelligent tab completion and full documentation in the form of UNIX manpages.REST APIProxmox VE uses a RESTful API. We choose JSON as primary data format, and the whole API is for-mally defined using JSON Schema. This enables fast and easy integration for third party managementtools like custom hosting environments.Role-based AdministrationYou can define granular access for all objects (like VMs, storages, nodes, etc.) by using the role baseduser- and permission management. This allows you to define privileges and helps you to controlaccess to objects. This concept is also known as access control lists: Each permission specifies asubject (a user or group) and a role (set of privileges) on a specific path.Authentication RealmsProxmox VE supports multiple authentication sources like Microsoft Active Directory, LDAP, Linux PAMstandard authentication or the built-in Proxmox VE authentication server. Proxmox VE Administration Guide 137 / 328NoteRequires option(s): ipgw6=<GatewayIPv6>Default gateway for IPv6 traffic.NoteRequires option(s): ip6ip=<IPv4Format/CIDR> (default = dhcp)IPv4 address in CIDR format.ip6=<IPv6Format/CIDR> (default = dhcp)IPv6 address in CIDR format.keyboard: <da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be |fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt |pt-br | sl | sv | tr>Keybord layout for vnc server. Default is read from the /etc/pve/datacenter.conf configuration file.Itshould not be necessary to set it.kvm: <boolean> (default = 1)Enable/disable KVM hardware virtualization.localtime: <boolean>Set the real time clock to local time. This is enabled by default if ostype indicates a Microsoft OS.lock: <backup | migrate | rollback | snapshot>Lock/unlock the VM.machine: (pc|pc(-i440fx)?-\d+\.\d+(\.pxe)?|q35|pc-q35-\d+\.\d+(\.pxe)?)Specific the Qemu machine type.memory: <integer> (16 - N) (default = 512)Amount of RAM for the VM in MB. This is the maximum available memory when you use the balloondevice.migrate_downtime: <number> (0 - N) (default = 0.1)Set maximum tolerated downtime (in seconds) for migrations.migrate_speed: <integer> (0 - N) (default = 0)Set maximum speed (in MB/s) for migrations. Value 0 is no limit.name: <string>Set a name for the VM. Only used on the configuration web interface. Proxmox VE Administration Guide 138 / 328nameserver: <string>cloud-init: Sets DNS server IP address for a container. Create will automatically use the setting fromthe host if neither searchdomain nor nameserver are set.net[n]: [model=]<enum> [,bridge=<bridge>] [,firewall=<1|0>][,link_down=<1|0>] [,macaddr=<XX:XX:XX:XX:XX:XX>] [,queues=<integer>] [,rate=<number>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,<model>=<macaddr>]Specify network devices.bridge=<bridge>Bridge to attach the network device to. The Proxmox VE standard bridge is called vmbr0.If you do not specify a bridge, we create a kvm user (NATed) network device, which providesDHCP and DNS services. The following addresses are used:10.0.2.2 Gateway10.0.2.3 DNS Server10.0.2.4 SMB ServerThe DHCP server assign addresses to the guest starting from 10.0.2.15.firewall=<boolean>Whether this interface should be protected by the firewall.link_down=<boolean>Whether this interface should be disconnected (like pulling the plug).macaddr=<XX:XX:XX:XX:XX:XX>MAC address. That address must be unique withing your network. This is automatically gener-ated if not specified.model=<e1000 | e1000-82540em | e1000-82544gc | e1000-82545em |i82551 | i82557b | i82559er | ne2k_isa | ne2k_pci | pcnet |rtl8139 | virtio | vmxnet3>Network Card Model. The virtio model provides the best performance with very low CPU over-head. If your guest does not support this driver, it is usually best to use e1000.queues=<integer> (0 - 16)Number of packet queues to be used on the device.rate=<number> (0 - N)Rate limit in mbps (megabytes per second) as floating point number.tag=<integer> (1 - 4094)VLAN tag to apply to packets on this interface.trunks=<vlanid[;vlanid...]>VLAN trunks to pass through this interface.numa: <boolean> (default = 0)Enable/disable NUMA. Proxmox VE Administration Guide 139 / 328numa[n]: cpus=<id[-id];...> [,hostnodes=<id[-id];...>] [,memory=<number>] [,policy=<preferred|bind|interleave>]NUMA topology.cpus=<id[-id];...>CPUs accessing this NUMA node.hostnodes=<id[-id];...>Host NUMA nodes to use.memory=<number>Amount of memory this NUMA node provides.policy=<bind | interleave | preferred>NUMA allocation policy.onboot: <boolean> (default = 0)Specifies whether a VM will be started during system bootup.ostype: <l24 | l26 | other | solaris | w2k | w2k3 | w2k8 | win10 |win7 | win8 | wvista | wxp>Specify guest operating system. This is used to enable special optimization/features for specific oper-ating systems:other unspecified OSwxp Microsoft Windows XPw2k Microsoft Windows 2000w2k3 Microsoft Windows 2003w2k8 Microsoft Windows 2008wvista Microsoft Windows Vistawin7 Microsoft Windows 7win8 Microsoft Windows 8/2012/2012r2win10 Microsoft Windows 10/2016l24 Linux 2.4 Kernell26 Linux 2.6/3.X Kernelsolaris Solaris/OpenSolaris/OpenIndiania kernel Proxmox VE Administration Guide 140 / 328parallel[n]: /dev/parport\d+|/dev/usb/lp\d+Map host parallel devices (n is 0 to 2).NoteThis option allows direct access to host hardware. So it is no longer possible to migrate suchmachines - use with special care.CautionExperimental! User reported problems with this option.protection: <boolean> (default = 0)Sets the protection flag of the VM. This will disable the remove VM and remove disk operations.reboot: <boolean> (default = 1)Allow reboot. If set to 0 the VM exit on reboot.sata[n]: [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,mbps=<mbps>] [,mbps_max=<mbps>] [,mbps_rd=<mbps>][,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>] [,mbps_wr_max=<mbps>][,media=<cdrom|disk>] [,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as SATA hard disk or CD-ROM (n is 0 to 5).aio=<native | threads>AIO type to use.backup=<boolean>Whether the drive should be included when making backups.bps=<bps>Maximum r/w speed in bytes per second.bps_max_length=<seconds>Maximum length of I/O bursts in seconds. Proxmox VE Administration Guide 141 / 328bps_rd=<bps>Maximum read speed in bytes per second.bps_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.bps_wr=<bps>Maximum write speed in bytes per second.bps_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.cache=<directsync | none | unsafe | writeback | writethrough>The drive’s cache modecyls=<integer>Force the drive’s physical geometry to have a specific cylinder count.detect_zeroes=<boolean>Controls whether to detect and try to optimize writes of zeroes.discard=<ignore | on>Controls whether to pass discard/trim requests to the underlying storage.file=<volume>The drive’s backing volume.format=<cloop | cow | qcow | qcow2 | qed | raw | vmdk>The drive’s backing file’s data format.heads=<integer>Force the drive’s physical geometry to have a specific head count.iops=<iops>Maximum r/w I/O in operations per second.iops_max=<iops>Maximum unthrottled r/w I/O pool in operations per second.iops_max_length=<seconds>Maximum length of I/O bursts in seconds.iops_rd=<iops>Maximum read I/O in operations per second.iops_rd_max=<iops>Maximum unthrottled read I/O pool in operations per second.iops_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.iops_wr=<iops>Maximum write I/O in operations per second. Proxmox VE Administration Guide 142 / 328iops_wr_max=<iops>Maximum unthrottled write I/O pool in operations per second.iops_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.mbps=<mbps>Maximum r/w speed in megabytes per second.mbps_max=<mbps>Maximum unthrottled r/w pool in megabytes per second.mbps_rd=<mbps>Maximum read speed in megabytes per second.mbps_rd_max=<mbps>Maximum unthrottled read pool in megabytes per second.mbps_wr=<mbps>Maximum write speed in megabytes per second.mbps_wr_max=<mbps>Maximum unthrottled write pool in megabytes per second.media=<cdrom | disk> (default = disk)The drive’s media type.replicate=<boolean> (default = 1)Whether the drive should considered for replication jobs.rerror=<ignore | report | stop>Read error action.secs=<integer>Force the drive’s physical geometry to have a specific sector count.serial=<serial>The drive’s reported serial number, url-encoded, up to 20 bytes long.shared=<boolean> (default = 0)Mark this locally-managed volume as available on all nodes.WarningThis option does not share the volume automatically, it assumes it is shared already!size=<DiskSize>Disk size. This is purely informational and has no effect.snapshot=<boolean>Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporaryand will be discarded when the VM is shutdown. Proxmox VE Administration Guide 143 / 328trans=<auto | lba | none>Force disk geometry bios translation mode.werror=<enospc | ignore | report | stop>Write error action.scsi[n]: [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,iothread=<1|0>] [,mbps=<mbps>] [,mbps_max=<mbps>][,mbps_rd=<mbps>] [,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>][,mbps_wr_max=<mbps>] [,media=<cdrom|disk>] [,queues=<integer>][,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,scsiblock=<1|0>] [,secs=<integer>] [,serial=<serial>] [,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as SCSI hard disk or CD-ROM (n is 0 to 13).aio=<native | threads>AIO type to use.backup=<boolean>Whether the drive should be included when making backups.bps=<bps>Maximum r/w speed in bytes per second.bps_max_length=<seconds>Maximum length of I/O bursts in seconds.bps_rd=<bps>Maximum read speed in bytes per second.bps_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.bps_wr=<bps>Maximum write speed in bytes per second.bps_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.cache=<directsync | none | unsafe | writeback | writethrough>The drive’s cache mode Proxmox VE Administration Guide 144 / 328cyls=<integer>Force the drive’s physical geometry to have a specific cylinder count.detect_zeroes=<boolean>Controls whether to detect and try to optimize writes of zeroes.discard=<ignore | on>Controls whether to pass discard/trim requests to the underlying storage.file=<volume>The drive’s backing volume.format=<cloop | cow | qcow | qcow2 | qed | raw | vmdk>The drive’s backing file’s data format.heads=<integer>Force the drive’s physical geometry to have a specific head count.iops=<iops>Maximum r/w I/O in operations per second.iops_max=<iops>Maximum unthrottled r/w I/O pool in operations per second.iops_max_length=<seconds>Maximum length of I/O bursts in seconds.iops_rd=<iops>Maximum read I/O in operations per second.iops_rd_max=<iops>Maximum unthrottled read I/O pool in operations per second.iops_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.iops_wr=<iops>Maximum write I/O in operations per second.iops_wr_max=<iops>Maximum unthrottled write I/O pool in operations per second.iops_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.iothread=<boolean>Whether to use iothreads for this drivembps=<mbps>Maximum r/w speed in megabytes per second.mbps_max=<mbps>Maximum unthrottled r/w pool in megabytes per second. Proxmox VE Administration Guide 145 / 328mbps_rd=<mbps>Maximum read speed in megabytes per second.mbps_rd_max=<mbps>Maximum unthrottled read pool in megabytes per second.mbps_wr=<mbps>Maximum write speed in megabytes per second.mbps_wr_max=<mbps>Maximum unthrottled write pool in megabytes per second.media=<cdrom | disk> (default = disk)The drive’s media type.queues=<integer> (2 - N)Number of queues.replicate=<boolean> (default = 1)Whether the drive should considered for replication jobs.rerror=<ignore | report | stop>Read error action.scsiblock=<boolean> (default = 0)whether to use scsi-block for full passthrough of host block deviceWarningcan lead to I/O errors in combination with low memory or high memory fragmentationon hostsecs=<integer>Force the drive’s physical geometry to have a specific sector count.serial=<serial>The drive’s reported serial number, url-encoded, up to 20 bytes long.shared=<boolean> (default = 0)Mark this locally-managed volume as available on all nodes.WarningThis option does not share the volume automatically, it assumes it is shared already!size=<DiskSize>Disk size. This is purely informational and has no effect.snapshot=<boolean>Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporaryand will be discarded when the VM is shutdown. Proxmox VE Administration Guide 146 / 328trans=<auto | lba | none>Force disk geometry bios translation mode.werror=<enospc | ignore | report | stop>Write error action.scsihw: <lsi | lsi53c810 | megasas | pvscsi | virtio-scsi-pci |virtio-scsi-single> (default = lsi)SCSI controller modelsearchdomain: <string>cloud-init: Sets DNS search domains for a container. Create will automatically use the setting from thehost if neither searchdomain nor nameserver are set.serial[n]: (/dev/.+|socket)Create a serial device inside the VM (n is 0 to 3), and pass through a host serial device (i.e. /dev/ttyS0),or create a unix socket on the host side (use qm terminal to open a terminal connection).NoteIf you pass through a host serial device, it is no longer possible to migrate such machines - use withspecial care.CautionExperimental! User reported problems with this option.shares: <integer> (0 - 50000) (default = 1000)Amount of memory shares for auto-ballooning. The larger the number is, the more memory this VMgets. Number is relative to weights of all other running VMs. Using zero disables auto-ballooningsmbios1: [family=<string>] [,manufacturer=<string>] [,product=<string>] [,serial=<string>] [,sku=<string>] [,uuid=<UUID>][,version=<string>]Specify SMBIOS type 1 fields.family=<string>Set SMBIOS1 family string.manufacturer=<string>Set SMBIOS1 manufacturer.product=<string>Set SMBIOS1 product ID.serial=<string>Set SMBIOS1 serial number. Proxmox VE Administration Guide 3 / 3281.2 Flexible StorageThe Proxmox VE storage model is very flexible. Virtual machine images can either be stored on one orseveral local storages or on shared storage like NFS and on SAN. There are no limits, you may configure asmany storage definitions as you like. You can use all storage technologies available for Debian Linux.One major benefit of storing VMs on shared storage is the ability to live-migrate running machines withoutany downtime, as all nodes in the cluster have direct access to VM disk images.We currently support the following Network storage types:• LVM Group (network backing with iSCSI targets)• iSCSI target• NFS Share• CIFS Share• Ceph RBD• Directly use iSCSI LUNs• GlusterFSLocal storage types supported are:• LVM Group (local backing devices like block devices, FC devices, DRBD, etc.)• Directory (storage on existing filesystem)• ZFS1.3 Integrated Backup and RestoreThe integrated backup tool (vzdump) creates consistent snapshots of running Containers and KVM guests.It basically creates an archive of the VM or CT data which includes the VM/CT configuration files.KVM live backup works for all storage types including VM images on NFS, CIFS, iSCSI LUN, Ceph RBD orSheepdog. The new backup format is optimized for storing VM backups fast and effective (sparse files, outof order data, minimized I/O).1.4 High Availability ClusterA multi-node Proxmox VE HA Cluster enables the definition of highly available virtual servers. The ProxmoxVE HA Cluster is based on proven Linux HA technologies, providing stable and reliable HA services. Proxmox VE Administration Guide 147 / 328sku=<string>Set SMBIOS1 SKU string.uuid=<UUID>Set SMBIOS1 UUID.version=<string>Set SMBIOS1 version.smp: <integer> (1 - N) (default = 1)The number of CPUs. Please use option -sockets instead.sockets: <integer> (1 - N) (default = 1)The number of CPU sockets.sshkeys: <string>cloud-init: Setup public SSH keys (one key per line, OpenSSH format).startdate: (now | YYYY-MM-DD | YYYY-MM-DDTHH:MM:SS) (default = now)Set the initial date of the real time clock. Valid format for date are: now or 2006-06-17T16:01:21 or2006-06-17.startup: `[[order=]\d+] [,up=\d+] [,down=\d+] `Startup and shutdown behavior. Order is a non-negative number defining the general startup order.Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds,which specifies a delay to wait before the next VM is started or stopped.tablet: <boolean> (default = 1)Enable/disable the USB tablet device. This device is usually needed to allow absolute mouse posi-tioning with VNC. Else the mouse runs out of sync with normal VNC clients. If you’re running lots ofconsole-only guests on one host, you may consider disabling this to save some context switches. Thisis turned off by default if you use spice (-vga=qxl).tdf: <boolean> (default = 0)Enable/disable time drift fix.template: <boolean> (default = 0)Enable/disable Template.unused[n]: <string>Reference to unused volumes. This is used internally, and should not be modified manually.usb[n]: [host=]<HOSTUSBDEVICE|spice> [,usb3=<1|0>]Configure an USB device (n is 0 to 4).host=<HOSTUSBDEVICE|spice>The Host USB device or port or the value spice. HOSTUSBDEVICE syntax is: Proxmox VE Administration Guide 148 / 328’bus-port(.port)*’ (decimal numbers) or’vendor_id:product_id’ (hexadeciaml numbers) or’spice’You can use the lsusb -t command to list existing usb devices.NoteThis option allows direct access to host hardware. So it is no longer possible to migrate suchmachines - use with special care.The value spice can be used to add a usb redirection devices for spice.usb3=<boolean> (default = 0)Specifies whether if given host option is a USB3 device or port (this does currently not workreliably with spice redirection and is then ignored).vcpus: <integer> (1 - N) (default = 0)Number of hotplugged vcpus.vga: <cirrus | qxl | qxl2 | qxl3 | qxl4 | serial0 | serial1 |serial2 | serial3 | std | vmware>Select the VGA type. If you want to use high resolution modes (>= 1280x1024x16) then you should usethe options std or vmware. Default is std for win8/win7/w2k8, and cirrus for other OS types. The qxloption enables the SPICE display sever. For win* OS you can select how many independent displaysyou want, Linux guests can add displays them self. You can also run without any graphic card, usinga serial device as terminal.virtio[n]: [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,iothread=<1|0>] [,mbps=<mbps>] [,mbps_max=<mbps>][,mbps_rd=<mbps>] [,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>][,mbps_wr_max=<mbps>] [,media=<cdrom|disk>] [,replicate=<1|0>][,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as VIRTIO hard disk (n is 0 to 15).aio=<native | threads>AIO type to use.backup=<boolean>Whether the drive should be included when making backups. Proxmox VE Administration Guide 149 / 328bps=<bps>Maximum r/w speed in bytes per second.bps_max_length=<seconds>Maximum length of I/O bursts in seconds.bps_rd=<bps>Maximum read speed in bytes per second.bps_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.bps_wr=<bps>Maximum write speed in bytes per second.bps_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.cache=<directsync | none | unsafe | writeback | writethrough>The drive’s cache modecyls=<integer>Force the drive’s physical geometry to have a specific cylinder count.detect_zeroes=<boolean>Controls whether to detect and try to optimize writes of zeroes.discard=<ignore | on>Controls whether to pass discard/trim requests to the underlying storage.file=<volume>The drive’s backing volume.format=<cloop | cow | qcow | qcow2 | qed | raw | vmdk>The drive’s backing file’s data format.heads=<integer>Force the drive’s physical geometry to have a specific head count.iops=<iops>Maximum r/w I/O in operations per second.iops_max=<iops>Maximum unthrottled r/w I/O pool in operations per second.iops_max_length=<seconds>Maximum length of I/O bursts in seconds.iops_rd=<iops>Maximum read I/O in operations per second.iops_rd_max=<iops>Maximum unthrottled read I/O pool in operations per second. Proxmox VE Administration Guide 150 / 328iops_rd_max_length=<seconds>Maximum length of read I/O bursts in seconds.iops_wr=<iops>Maximum write I/O in operations per second.iops_wr_max=<iops>Maximum unthrottled write I/O pool in operations per second.iops_wr_max_length=<seconds>Maximum length of write I/O bursts in seconds.iothread=<boolean>Whether to use iothreads for this drivembps=<mbps>Maximum r/w speed in megabytes per second.mbps_max=<mbps>Maximum unthrottled r/w pool in megabytes per second.mbps_rd=<mbps>Maximum read speed in megabytes per second.mbps_rd_max=<mbps>Maximum unthrottled read pool in megabytes per second.mbps_wr=<mbps>Maximum write speed in megabytes per second.mbps_wr_max=<mbps>Maximum unthrottled write pool in megabytes per second.media=<cdrom | disk> (default = disk)The drive’s media type.replicate=<boolean> (default = 1)Whether the drive should considered for replication jobs.rerror=<ignore | report | stop>Read error action.secs=<integer>Force the drive’s physical geometry to have a specific sector count.serial=<serial>The drive’s reported serial number, url-encoded, up to 20 bytes long.shared=<boolean> (default = 0)Mark this locally-managed volume as available on all nodes. Proxmox VE Administration Guide 151 / 328WarningThis option does not share the volume automatically, it assumes it is shared already!size=<DiskSize>Disk size. This is purely informational and has no effect.snapshot=<boolean>Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporaryand will be discarded when the VM is shutdown.trans=<auto | lba | none>Force disk geometry bios translation mode.werror=<enospc | ignore | report | stop>Write error action.vmstatestorage: <string>Default storage for VM state volumes/files.watchdog: [[model=]<i6300esb|ib700>] [,action=<enum>]Create a virtual hardware watchdog device. Once enabled (by a guest action), the watchdog must beperiodically polled by an agent inside the guest or else the watchdog will reset the guest (or executethe respective action specified)action=<debug | none | pause | poweroff | reset | shutdown>The action to perform if after activation the guest fails to poll the watchdog in time.model=<i6300esb | ib700> (default = i6300esb)Watchdog type to emulate.10.10 LocksOnline migrations, snapshots and backups (vzdump) set a lock to prevent incompatible concurrent actionson the affected VMs. Sometimes you need to remove such a lock manually (e.g., after a power failure).qm unlock <vmid>CautionOnly do that if you are sure the action which set the lock is no longer running. Proxmox VE Administration Guide 152 / 328Chapter 11Proxmox Container ToolkitContainers are a lightweight alternative to fully virtualized VMs. Instead of emulating a complete OperatingSystem (OS), containers simply use the OS of the host they run on. This implies that all containers use thesame kernel, and that they can access resources from the host directly.This is great because containers do not waste CPU power nor memory due to kernel emulation. Containerrun-time costs are close to zero and usually negligible. But there are also some drawbacks you need toconsider:• You can only run Linux based OS inside containers, i.e. it is not possible to run FreeBSD or MS Windowsinside.• For security reasons, access to host resources needs to be restricted. This is done with AppArmor, Sec-Comp filters and other kernel features. Be prepared that some syscalls are not allowed inside containers.Proxmox VE uses LXC as underlying container technology. We consider LXC as low-level library, whichprovides countless options. It would be too difficult to use those tools directly. Instead, we provide a smallwrapper called pct, the "Proxmox Container Toolkit".The toolkit is tightly coupled with Proxmox VE. That means that it is aware of the cluster setup, and it canuse the same network and storage resources as fully virtualized VMs. You can even use the Proxmox VEfirewall, or manage containers using the HA framework.Our primary goal is to offer an environment as one would get from a VM, but without the additional overhead.We call this "System Containers".NoteIf you want to run micro-containers (with docker, rkt, . . . ), it is best to run them inside a VM.11.1 Technology Overview• LXC (https://linuxcontainers.org/)• Integrated into Proxmox VE graphical user interface (GUI)• Easy to use command line tool pct Proxmox VE Administration Guide 153 / 328• Access via Proxmox VE REST API• lxcfs to provide containerized /proc file system• AppArmor/Seccomp to improve security• CRIU: for live migration (planned)• Use latest available kernels (4.4.X)• Image based deployment (templates)• Use Proxmox VE storage library• Container setup from host (network, DNS, storage, . . . )11.2 Security ConsiderationsContainers use the same kernel as the host, so there is a big attack surface for malicious users. You shouldconsider this fact if you provide containers to totally untrusted people. In general, fully virtualized VMs providebetter isolation.The good news is that LXC uses many kernel security features like AppArmor, CGroups and PID and usernamespaces, which makes containers usage quite secure.11.3 Guest Operating System ConfigurationWe normally try to detect the operating system type inside the container, and then modify some files insidethe container to make them work as expected. Here is a short list of things we do at container startup:set /etc/hostnameto set the container namemodify /etc/hoststo allow lookup of the local hostnamenetwork setuppass the complete network setup to the containerconfigure DNSpass information about DNS serversadapt the init systemfor example, fix the number of spawned getty processesset the root passwordwhen creating a new container Proxmox VE Administration Guide 154 / 328rewrite ssh_host_keysso that each container has unique keysrandomize crontabso that cron does not start at the same time on all containersChanges made by Proxmox VE are enclosed by comment markers:# --- BEGIN PVE ---<data># --- END PVE ---Those markers will be inserted at a reasonable location in the file. If such a section already exists, it will beupdated in place and will not be moved.Modification of a file can be prevented by adding a .pve-ignore. file for it. For instance, if the file /etc/.pve-ignore.hosts exists then the /etc/hosts file will not be touched. This can be a simpleempty file created via:# touch /etc/.pve-ignore.hostsMost modifications are OS dependent, so they differ between different distributions and versions. You cancompletely disable modifications by manually setting the ostype to unmanaged.OS type detection is done by testing for certain files inside the container:Ubuntuinspect /etc/lsb-release (DISTRIB_ID=Ubuntu)Debiantest /etc/debian_versionFedoratest /etc/fedora-releaseRedHat or CentOStest /etc/redhat-releaseArchLinuxtest /etc/arch-releaseAlpinetest /etc/alpine-releaseGentootest /etc/gentoo-releaseNoteContainer start fails if the configured ostype differs from the auto detected type. Proxmox VE Administration Guide 155 / 32811.4 Container ImagesContainer images, sometimes also referred to as “templates” or “appliances”, are tar archives which containeverything to run a container. You can think of it as a tidy container backup. Like most modern containertoolkits, pct uses those images when you create a new container, for example:pct create 999 local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gzProxmox VE itself ships a set of basic templates for most common operating systems, and you can downloadthem using the pveam (short for Proxmox VE Appliance Manager) command line utility. You can alsodownload TurnKey Linux containers using that tool (or the graphical user interface).Our image repositories contain a list of available images, and there is a cron job run each day to downloadthat list. You can trigger that update manually with:pveam updateAfter that you can view the list of available images using:pveam availableYou can restrict this large list by specifying the section you are interested in, for example basic systemimages:List available system images# pveam available --section systemsystem archlinux-base_2015-24-29-1_x86_64.tar.gzsystem centos-7-default_20160205_amd64.tar.xzsystem debian-6.0-standard_6.0-7_amd64.tar.gzsystem debian-7.0-standard_7.0-3_amd64.tar.gzsystem debian-8.0-standard_8.0-1_amd64.tar.gzsystem ubuntu-12.04-standard_12.04-1_amd64.tar.gzsystem ubuntu-14.04-standard_14.04-1_amd64.tar.gzsystem ubuntu-15.04-standard_15.04-1_amd64.tar.gzsystem ubuntu-15.10-standard_15.10-1_amd64.tar.gzBefore you can use such a template, you need to download them into one of your storages. You can simplyuse storage local for that purpose. For clustered installations, it is preferred to use a shared storage sothat all nodes can access those images.pveam download local debian-8.0-standard_8.0-1_amd64.tar.gzYou are now ready to create containers using that image, and you can list all downloaded images on storagelocal with:# pveam list locallocal:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz 190.20MBThe above command shows you the full Proxmox VE volume identifiers. They include the storage name, andmost other Proxmox VE commands can use them. For example you can delete that image later with:pveam remove local:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz Proxmox VE Administration Guide 156 / 32811.5 Container StorageTraditional containers use a very simple storage model, only allowing a single mount point, the root filesystem. This was further restricted to specific file system types like ext4 and nfs. Additional mounts areoften done by user provided scripts. This turned out to be complex and error prone, so we try to avoid thatnow.Our new LXC based container model is more flexible regarding storage. First, you can have more than asingle mount point. This allows you to choose a suitable storage for each application. For example, you canuse a relatively slow (and thus cheap) storage for the container root file system. Then you can use a secondmount point to mount a very fast, distributed storage for your database application. See section Mount Pointsfor further details.The second big improvement is that you can use any storage type supported by the Proxmox VE storagelibrary. That means that you can store your containers on local lvmthin or zfs, shared iSCSI storage,or even on distributed storage systems like ceph. It also enables us to use advanced storage features likesnapshots and clones. vzdump can also use the snapshot feature to provide consistent container backups.Last but not least, you can also mount local devices directly, or mount local directories using bind mounts.That way you can access local storage inside containers with zero overhead. Such bind mounts also providean easy way to share data between different containers.11.5.1 FUSE MountsWarningBecause of existing issues in the Linux kernel’s freezer subsystem the usage of FUSE mounts insidea container is strongly advised against, as containers need to be frozen for suspend or snapshotmode backups.If FUSE mounts cannot be replaced by other mounting mechanisms or storage technologies, it is possible toestablish the FUSE mount on the Proxmox host and use a bind mount point to make it accessible inside thecontainer.11.5.2 Using Quotas Inside ContainersQuotas allow to set limits inside a container for the amount of disk space that each user can use. This onlyworks on ext4 image based storage types and currently does not work with unprivileged containers.Activating the quota option causes the following mount options to be used for a mount point: usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0This allows quotas to be used like you would on any other system. You can initialize the /aquota.userand /aquota.group files by runningquotacheck -cmug /quotaon /and edit the quotas via the edquota command. Refer to the documentation of the distribution runninginside the container for details. Proxmox VE Administration Guide 4 / 3281.5 Flexible NetworkingProxmox VE uses a bridged networking model. All VMs can share one bridge as if virtual network cablesfrom each guest were all plugged into the same switch. For connecting VMs to the outside world, bridgesare attached to physical network cards assigned a TCP/IP configuration.For further flexibility, VLANs (IEEE 802.1q) and network bonding/aggregation are possible. In this way it ispossible to build complex, flexible virtual networks for the Proxmox VE hosts, leveraging the full power of theLinux network stack.1.6 Integrated FirewallThe integrated firewall allows you to filter network packets on any VM or Container interface. Common setsof firewall rules can be grouped into “security groups”.1.7 Why Open SourceProxmox VE uses a Linux kernel and is based on the Debian GNU/Linux Distribution. The source code ofProxmox VE is released under the GNU Affero General Public License, version 3. This means that you arefree to inspect the source code at any time or contribute to the project yourself.At Proxmox we are committed to use open source software whenever possible. Using open source softwareguarantees full access to all functionalities - as well as high security and reliability. We think that everybodyshould have the right to access the source code of a software to run it, build on it, or submit changes backto the project. Everybody is encouraged to contribute while Proxmox ensures the product always meetsprofessional quality criteria.Open source software also helps to keep your costs low and makes your core infrastructure independentfrom a single vendor.1.8 Your benefit with Proxmox VE• Open source software• No vendor lock-in• Linux kernel• Fast installation and easy-to-use• Web-based management interface• REST API• Huge active community• Low administration costs and simple deployment Proxmox VE Administration Guide 157 / 328NoteYou need to run the above commands for every mount point by passing the mount point’s path instead ofjust /.11.5.3 Using ACLs Inside ContainersThe standard Posix Access Control Lists are also available inside containers. ACLs allow you to set moredetailed file ownership than the traditional user/ group/others model.11.5.4 Backup of Containers mount pointsBy default additional mount points besides the Root Disk mount point are not included in backups. You canreverse this default behavior by setting the Backup option on a mount point.11.5.5 Replication of Containers mount pointsBy default additional mount points are replicated when the Root Disk is replicated. If you want the ProxmoxVE storage replication mechanism to skip a mount point when starting a replication job, you can set the Skipreplication option on that mount point.As of Proxmox VE 5.0, replication requires a storage of type zfspool, so adding a mount point to adifferent type of storage when the container has replication configured requires to Skip replication for thatmount point.11.6 Container Settings11.6.1 General SettingsGeneral settings of a container include Proxmox VE Administration Guide 158 / 328• the Node : the physical server on which the container will run• the CT ID: a unique number in this Proxmox VE installation used to identify your container• Hostname: the hostname of the container• Resource Pool: a logical group of containers and VMs• Password: the root password of the container• SSH Public Key: a public key for connecting to the root account over SSH• Unprivileged container: this option allows to choose at creation time if you want to create a privileged orunprivileged container.Privileged ContainersSecurity is done by dropping capabilities, using mandatory access control (AppArmor), SecComp filters andnamespaces. The LXC team considers this kind of container as unsafe, and they will not consider newcontainer escape exploits to be security issues worthy of a CVE and quick fix. So you should use this kind ofcontainers only inside a trusted environment, or when no untrusted task is running as root in the container.Unprivileged ContainersThis kind of containers use a new kernel feature called user namespaces. The root UID 0 inside the containeris mapped to an unprivileged user outside the container. This means that most security issues (containerescape, resource abuse, . . . ) in those containers will affect a random unprivileged user, and so would be ageneric kernel security bug rather than an LXC issue. The LXC team thinks unprivileged containers are safeby design.NoteIf the container uses systemd as an init system, please be aware the systemd version running inside thecontainer should be equal or greater than 220. Proxmox VE Administration Guide 159 / 32811.6.2 CPUYou can restrict the number of visible CPUs inside the container using the cores option. This is imple-mented using the Linux cpuset cgroup (control group). A special task inside pvestatd tries to distributerunning containers among available CPUs. You can view the assigned CPUs using the following command:# pct cpusets---------------------102: 6 7105: 2 3 4 5108: 0 1---------------------Containers use the host kernel directly, so all task inside a container are handled by the host CPU scheduler.Proxmox VE uses the Linux CFS (Completely Fair Scheduler) scheduler by default, which has additionalbandwidth control options.cpulimit: You can use this option to further limit assigned CPU time. Please note that this is afloating point number, so it is perfectly valid to assign two cores to a container, butrestrict overall CPU consumption to half a core.cores: 2cpulimit: 0.5cpuunits: This is a relative weight passed to the kernel scheduler. The larger the number is, themore CPU time this container gets. Number is relative to the weights of all the otherrunning containers. The default is 1024. You can use this setting to prioritize somecontainers. Proxmox VE Administration Guide 160 / 32811.6.3 MemoryContainer memory is controlled using the cgroup memory controller.memory: Limit overall memory usage. This corresponds to the memory.limit_in_bytescgroup setting.swap: Allows the container to use additional swap memory from the host swap space. Thiscorresponds to the memory.memsw.limit_in_bytes cgroup setting, which isset to the sum of both value (memory + swap).11.6.4 Mount Points Proxmox VE Administration Guide 161 / 328The root mount point is configured with the rootfs property, and you can configure up to 10 additionalmount points. The corresponding options are called mp0 to mp9, and they can contain the following setting:rootfs: [volume=]<volume> [,acl=<1|0>] [,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>] [,size=<DiskSize>]Use volume as container root. See below for a detailed description of all options.mp[n]: [volume=]<volume> ,mp=<Path> [,acl=<1|0>] [,backup=<1|0>][,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>][,size=<DiskSize>]Use volume as container mount point.acl=<boolean>Explicitly enable or disable ACL support.backup=<boolean>Whether to include the mount point in backups (only used for volume mount points).mp=<Path>Path to the mount point as seen from inside the container.NoteMust not contain any symlinks for security reasons.quota=<boolean>Enable user quotas inside the container (not supported with zfs subvolumes)replicate=<boolean> (default = 1)Will include this volume to a storage replica job.ro=<boolean>Read-only mount pointshared=<boolean> (default = 0)Mark this non-volume mount point as available on all nodes.WarningThis option does not share the mount point automatically, it assumes it is shared al-ready!size=<DiskSize>Volume size (read only value).volume=<volume>Volume, device or directory to mount into the container.Currently there are basically three types of mount points: storage backed mount points, bind mounts anddevice mounts. Proxmox VE Administration Guide 162 / 328Typical container rootfs configurationrootfs: thin1:base-100-disk-1,size=8GStorage Backed Mount PointsStorage backed mount points are managed by the Proxmox VE storage subsystem and come in three differ-ent flavors:• Image based: these are raw images containing a single ext4 formatted file system.• ZFS subvolumes: these are technically bind mounts, but with managed storage, and thus allow resizingand snapshotting.• Directories: passing size=0 triggers a special case where instead of a raw image a directory is created.NoteThe special option syntax STORAGE_ID:SIZE_IN_GB for storage backed mount point volumes willautomatically allocate a volume of the specified size on the specified storage. E.g., calling pct set100 -mp0 thin1:10,mp=/path/in/container will allocate a 10GB volume on the storagethin1 and replace the volume ID place holder 10 with the allocated volume ID.Bind Mount PointsBind mounts allow you to access arbitrary directories from your Proxmox VE host inside a container. Somepotential use cases are:• Accessing your home directory in the guest• Accessing an USB device directory in the guest• Accessing an NFS mount from the host in the guestBind mounts are considered to not be managed by the storage subsystem, so you cannot make snapshotsor deal with quotas from inside the container. With unprivileged containers you might run into permissionproblems caused by the user mapping and cannot use ACLs.NoteThe contents of bind mount points are not backed up when using vzdump.WarningFor security reasons, bind mounts should only be established using source directories especiallyreserved for this purpose, e.g., a directory hierarchy under /mnt/bindmounts. Never bindmount system directories like /, /var or /etc into a container - this poses a great security risk. Proxmox VE Administration Guide 163 / 328NoteThe bind mount source path must not contain any symlinks.For example, to make the directory /mnt/bindmounts/shared accessible in the container with ID 100under the path /shared, use a configuration line like mp0: /mnt/bindmounts/shared,mp=/shared in /etc/pve/lxc/100.conf. Alternatively, use pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared to achieve the same result.Device Mount PointsDevice mount points allow to mount block devices of the host directly into the container. Similar to bindmounts, device mounts are not managed by Proxmox VE’s storage subsystem, but the quota and acloptions will be honored.NoteDevice mount points should only be used under special circumstances. In most cases a storage backedmount point offers the same performance and a lot more features.NoteThe contents of device mount points are not backed up when using vzdump.11.6.5 NetworkYou can configure up to 10 network interfaces for a single container. The corresponding options are callednet0 to net9, and they can contain the following setting: Proxmox VE Administration Guide 164 / 328net[n]: name=<string> [,bridge=<bridge>] [,firewall=<1|0>] [,gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,hwaddr=<XX:XX:XX:XX:XX:XX>][,ip=<(IPv4/CIDR|dhcp|manual)>] [,ip6=<(IPv6/CIDR|auto|dhcp|manual)>] [,mtu=<integer>] [,rate=<mbps>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,type=<veth>]Specifies network interfaces for the container.bridge=<bridge>Bridge to attach the network device to.firewall=<boolean>Controls whether this interface’s firewall rules should be used.gw=<GatewayIPv4>Default gateway for IPv4 traffic.gw6=<GatewayIPv6>Default gateway for IPv6 traffic.hwaddr=<XX:XX:XX:XX:XX:XX>The interface MAC address. This is dynamically allocated by default, but you can set that stati-cally if needed, for example to always have the same link-local IPv6 address. (lxc.network.hwaddr)ip=<(IPv4/CIDR|dhcp|manual)>IPv4 address in CIDR format.ip6=<(IPv6/CIDR|auto|dhcp|manual)>IPv6 address in CIDR format.mtu=<integer> (64 - N)Maximum transfer unit of the interface. (lxc.network.mtu)name=<string>Name of the network device as seen from inside the container. (lxc.network.name)rate=<mbps>Apply rate limiting to the interfacetag=<integer> (1 - 4094)VLAN tag for this interface.trunks=<vlanid[;vlanid...]>VLAN ids to pass through the interfacetype=<veth>Network interface type.11.6.6 Automatic Start and Shutdown of ContainersAfter creating your containers, you probably want them to start automatically when the host system boots.For this you need to select the option Start at boot from the Options Tab of your container in the web interface,or set it with the following command: Proxmox VE Administration Guide 165 / 328pct set <ctid> -onboot 1Start and Shutdown OrderIf you want to fine tune the boot order of your containers, you can use the following parameters :• Start/Shutdown order: Defines the start order priority. E.g. set it to 1 if you want the CT to be the first tobe started. (We use the reverse startup order for shutdown, so a container with a start order of 1 would bethe last to be shut down)• Startup delay: Defines the interval between this container start and subsequent containers starts . E.g.set it to 240 if you want to wait 240 seconds before starting other containers.• Shutdown timeout: Defines the duration in seconds Proxmox VE should wait for the container to be offlineafter issuing a shutdown command. By default this value is set to 60, which means that Proxmox VE willissue a shutdown request, wait 60s for the machine to be offline, and if after 60s the machine is still onlinewill notify that the shutdown action failed.Please note that containers without a Start/Shutdown order parameter will always start after those where theparameter is set, and this parameter only makes sense between the machines running locally on a host, andnot cluster-wide.11.7 Backup and Restore11.7.1 Container BackupIt is possible to use the vzdump tool for container backup. Please refer to the vzdump manual page fordetails.11.7.2 Restoring Container BackupsRestoring container backups made with vzdump is possible using the pct restore command. Bydefault, pct restore will attempt to restore as much of the backed up container configuration as possible.It is possible to override the backed up configuration by manually setting container options on the commandline (see the pct manual page for details).Notepvesm extractconfig can be used to view the backed up configuration contained in a vzdumparchive.There are two basic restore modes, only differing by their handling of mount points: Proxmox VE Administration Guide 166 / 328“Simple” Restore ModeIf neither the rootfs parameter nor any of the optional mpX parameters are explicitly set, the mount pointconfiguration from the backed up configuration file is restored using the following steps:1. Extract mount points and their options from backup2. Create volumes for storage backed mount points (on storage provided with the storage parameter,or default local storage if unset)3. Extract files from backup archive4. Add bind and device mount points to restored configuration (limited to root user)NoteSince bind and device mount points are never backed up, no files are restored in the last step, but onlythe configuration options. The assumption is that such mount points are either backed up with anothermechanism (e.g., NFS space that is bind mounted into many containers), or not intended to be backed upat all.This simple mode is also used by the container restore operations in the web interface.“Advanced” Restore ModeBy setting the rootfs parameter (and optionally, any combination of mpX parameters), the pct restore command is automatically switched into an advanced mode. This advanced mode completely ignoresthe rootfs and mpX configuration options contained in the backup archive, and instead only uses theoptions explicitly provided as parameters.This mode allows flexible configuration of mount point settings at restore time, for example:• Set target storages, volume sizes and other options for each mount point individually• Redistribute backed up files according to new mount point scheme• Restore to device and/or bind mount points (limited to root user)11.8 Managing Containers with pctpct is the tool to manage Linux Containers on Proxmox VE. You can create and destroy containers, andcontrol execution (start, stop, migrate, . . . ). You can use pct to set parameters in the associated config file,like network configuration or memory limits. Proxmox VE Administration Guide 5 / 3281.9 Getting Help1.9.1 Proxmox VE WikiThe primary source of information is the Proxmox VE Wiki. It combines the reference documentation withuser contributed content.1.9.2 Community Support ForumProxmox VE itself is fully open source, so we always encourage our users to discuss and share their knowl-edge using the Proxmox VE Community Forum. The forum is fully moderated by the Proxmox support team,and has a quite large user base around the whole world. Needless to say that such a large forum is a greatplace to get information.1.9.3 Mailing ListsThis is a fast way to communicate via email with the Proxmox VE community• Mailing list for users: PVE User ListThe primary communication channel for developers is:• Mailing list for developer: PVE development discussion1.9.4 Commercial SupportProxmox Server Solutions Gmbh also offers commercial Proxmox VE Subscription Service Plans. SystemAdministrators with a standard subscription plan can access a dedicated support portal with guaranteed re-sponse time, where Proxmox VE developers help them should an issue appear. Please contact the Proxmoxsales team for more information or volume discounts.1.9.5 Bug TrackerWe also run a public bug tracker at https://bugzilla.proxmox.com. If you ever detect an issue, you can file abug report there. This makes it easy to track its status, and you will get notified as soon as the problem isfixed.1.10 Project HistoryThe project started in 2007, followed by a first stable version in 2008. At the time we used OpenVZ forcontainers, and KVM for virtual machines. The clustering features were limited, and the user interface wassimple (server generated web page).But we quickly developed new features using the Corosync cluster stack, and the introduction of the newProxmox cluster file system (pmxcfs) was a big step forward, because it completely hides the cluster com-plexity from the user. Managing a cluster of 16 nodes is as simple as managing a single node. Proxmox VE Administration Guide 167 / 32811.8.1 CLI Usage ExamplesCreate a container based on a Debian template (provided you have already downloaded the template viathe web interface)pct create 100 /var/lib/vz/template/cache/debian-8.0-standard_8.0-1 ←-_amd64.tar.gzStart container 100pct start 100Start a login session via gettypct console 100Enter the LXC namespace and run a shell as root userpct enter 100Display the configurationpct config 100Add a network interface called eth0, bridged to the host bridge vmbr0, set the address and gateway, whileit’s runningpct set 100 -net0 name=eth0,bridge=vmbr0,ip=192.168.15.147/24,gw ←-=192.168.15.1Reduce the memory of the container to 512MBpct set 100 -memory 51211.8.2 Obtaining Debugging LogsIn case pct start is unable to start a specific container, it might be helpful to collect debugging outputby running lxc-start (replace ID with the container’s ID):lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.logThis command will attempt to start the container in foreground mode, to stop the container run pct shutdown ID or pct stop ID in a second terminal.The collected debug log is written to /tmp/lxc-ID.log.NoteIf you have changed the container’s configuration since the last start attempt with pct start, you needto run pct start at least once to also update the configuration used by lxc-start. Proxmox VE Administration Guide 168 / 32811.9 MigrationIf you have a cluster, you can migrate your Containers withpct migrate <vmid> <target>This works as long as your Container is offline. If it has local volumes or mountpoints defined, the migrationwill copy the content over the network to the target host if there is the same storage defined.If you want to migrate online Containers, the only way is to use restart migration. This can be initiated withthe -restart flag and the optional -timeout parameter.A restart migration will shut down the Container and kill it after the specified timeout (the default is 180seconds). Then it will migrate the Container like an offline migration and when finished, it starts the Containeron the target node.11.10 ConfigurationThe /etc/pve/lxc/<CTID>.conf file stores container configuration, where <CTID> is the numericID of the given container. Like all other files stored inside /etc/pve/, they get automatically replicated toall other cluster nodes.NoteCTIDs < 100 are reserved for internal purposes, and CTIDs need to be unique cluster wide.Example Container Configurationostype: debianarch: amd64hostname: wwwmemory: 512swap: 512net0: bridge=vmbr0,hwaddr=66:64:66:64:64:36,ip=dhcp,name=eth0,type=vethrootfs: local:107/vm-107-disk-1.raw,size=7GThose configuration files are simple text files, and you can edit them using a normal text editor (vi, nano,. . . ). This is sometimes useful to do small corrections, but keep in mind that you need to restart the containerto apply such changes.For that reason, it is usually better to use the pct command to generate and modify those files, or do thewhole thing using the GUI. Our toolkit is smart enough to instantaneously apply most changes to runningcontainers. This feature is called "hot plug", and there is no need to restart the container in that case.11.10.1 File FormatContainer configuration files use a simple colon separated key/value format. Each line has the followingformat: Proxmox VE Administration Guide 169 / 328# this is a commentOPTION: valueBlank lines in those files are ignored, and lines starting with a # character are treated as comments and arealso ignored.It is possible to add low-level, LXC style configuration directly, for example:lxc.init_cmd: /sbin/my_own_initorlxc.init_cmd = /sbin/my_own_initThose settings are directly passed to the LXC low-level tools.11.10.2 SnapshotsWhen you create a snapshot, pct stores the configuration at snapshot time into a separate snapshot sec-tion within the same configuration file. For example, after creating a snapshot called “testsnapshot”, yourconfiguration file will look like this:Container configuration with snapshotmemory: 512swap: 512parent: testsnaphot...[testsnaphot]memory: 512swap: 512snaptime: 1457170803...There are a few snapshot related properties like parent and snaptime. The parent property is usedto store the parent/child relationship between snapshots. snaptime is the snapshot creation time stamp(Unix epoch).11.10.3 Optionsarch: <amd64 | i386> (default = amd64)OS architecture type.cmode: <console | shell | tty> (default = tty)Console mode. By default, the console command tries to open a connection to one of the availabletty devices. By setting cmode to console it tries to attach to /dev/console instead. If you set cmode toshell, it simply invokes a shell inside the container (no login). Proxmox VE Administration Guide 170 / 328console: <boolean> (default = 1)Attach a console device (/dev/console) to the container.cores: <integer> (1 - 128)The number of cores assigned to the container. A container can use all available cores by default.cpulimit: <number> (0 - 128) (default = 0)Limit of CPU usage.NoteIf the computer has 2 CPUs, it has a total of 2 CPU time. Value 0 indicates no CPU limit.cpuunits: <integer> (0 - 500000) (default = 1024)CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, themore CPU time this VM gets. Number is relative to the weights of all the other running VMs.NoteYou can disable fair-scheduler configuration by setting this to 0.description: <string>Container description. Only used on the configuration web interface.hostname: <string>Set a host name for the container.lock: <backup | migrate | rollback | snapshot>Lock/unlock the VM.memory: <integer> (16 - N) (default = 512)Amount of RAM for the VM in MB.mp[n]: [volume=]<volume> ,mp=<Path> [,acl=<1|0>] [,backup=<1|0>][,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>][,size=<DiskSize>]Use volume as container mount point.acl=<boolean>Explicitly enable or disable ACL support.backup=<boolean>Whether to include the mount point in backups (only used for volume mount points).mp=<Path>Path to the mount point as seen from inside the container. Proxmox VE Administration Guide 171 / 328NoteMust not contain any symlinks for security reasons.quota=<boolean>Enable user quotas inside the container (not supported with zfs subvolumes)replicate=<boolean> (default = 1)Will include this volume to a storage replica job.ro=<boolean>Read-only mount pointshared=<boolean> (default = 0)Mark this non-volume mount point as available on all nodes.WarningThis option does not share the mount point automatically, it assumes it is shared al-ready!size=<DiskSize>Volume size (read only value).volume=<volume>Volume, device or directory to mount into the container.nameserver: <string>Sets DNS server IP address for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.net[n]: name=<string> [,bridge=<bridge>] [,firewall=<1|0>] [,gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,hwaddr=<XX:XX:XX:XX:XX:XX>][,ip=<(IPv4/CIDR|dhcp|manual)>] [,ip6=<(IPv6/CIDR|auto|dhcp|manual)>] [,mtu=<integer>] [,rate=<mbps>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,type=<veth>]Specifies network interfaces for the container.bridge=<bridge>Bridge to attach the network device to.firewall=<boolean>Controls whether this interface’s firewall rules should be used.gw=<GatewayIPv4>Default gateway for IPv4 traffic.gw6=<GatewayIPv6>Default gateway for IPv6 traffic. Proxmox VE Administration Guide 172 / 328hwaddr=<XX:XX:XX:XX:XX:XX>The interface MAC address. This is dynamically allocated by default, but you can set that stati-cally if needed, for example to always have the same link-local IPv6 address. (lxc.network.hwaddr)ip=<(IPv4/CIDR|dhcp|manual)>IPv4 address in CIDR format.ip6=<(IPv6/CIDR|auto|dhcp|manual)>IPv6 address in CIDR format.mtu=<integer> (64 - N)Maximum transfer unit of the interface. (lxc.network.mtu)name=<string>Name of the network device as seen from inside the container. (lxc.network.name)rate=<mbps>Apply rate limiting to the interfacetag=<integer> (1 - 4094)VLAN tag for this interface.trunks=<vlanid[;vlanid...]>VLAN ids to pass through the interfacetype=<veth>Network interface type.onboot: <boolean> (default = 0)Specifies whether a VM will be started during system bootup.ostype: <alpine | archlinux | centos | debian | fedora | gentoo |opensuse | ubuntu | unmanaged>OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scriptsin /usr/share/lxc/config/<ostype>.common.conf. Value unmanaged can be used to skip and OS spe-cific setup.protection: <boolean> (default = 0)Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.rootfs: [volume=]<volume> [,acl=<1|0>] [,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>] [,size=<DiskSize>]Use volume as container root.acl=<boolean>Explicitly enable or disable ACL support.quota=<boolean>Enable user quotas inside the container (not supported with zfs subvolumes) Proxmox VE Administration Guide 173 / 328replicate=<boolean> (default = 1)Will include this volume to a storage replica job.ro=<boolean>Read-only mount pointshared=<boolean> (default = 0)Mark this non-volume mount point as available on all nodes.WarningThis option does not share the mount point automatically, it assumes it is shared al-ready!size=<DiskSize>Volume size (read only value).volume=<volume>Volume, device or directory to mount into the container.searchdomain: <string>Sets DNS search domains for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.startup: `[[order=]\d+] [,up=\d+] [,down=\d+] `Startup and shutdown behavior. Order is a non-negative number defining the general startup order.Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds,which specifies a delay to wait before the next VM is started or stopped.swap: <integer> (0 - N) (default = 512)Amount of SWAP for the VM in MB.template: <boolean> (default = 0)Enable/disable Template.tty: <integer> (0 - 6) (default = 2)Specify the number of tty available to the containerunprivileged: <boolean> (default = 0)Makes the container run as unprivileged user. (Should not be modified manually.)unused[n]: <string>Reference to unused volumes. This is used internally, and should not be modified manually. Proxmox VE Administration Guide 174 / 32811.11 LocksContainer migrations, snapshots and backups (vzdump) set a lock to prevent incompatible concurrent ac-tions on the affected container. Sometimes you need to remove such a lock manually (e.g., after a powerfailure).pct unlock <CTID>CautionOnly do that if you are sure the action which set the lock is no longer running. Proxmox VE Administration Guide 175 / 328Chapter 12Proxmox VE FirewallProxmox VE Firewall provides an easy way to protect your IT infrastructure. You can setup firewall rules forall hosts inside a cluster, or define rules for virtual machines and containers. Features like firewall macros,security groups, IP sets and aliases help to make that task easier.While all configuration is stored on the cluster file system, the iptables-based firewall runs on eachcluster node, and thus provides full isolation between virtual machines. The distributed nature of this systemalso provides much higher bandwidth than a central firewall solution.The firewall has full support for IPv4 and IPv6. IPv6 support is fully transparent, and we filter traffic for bothprotocols by default. So there is no need to maintain a different set of rules for IPv6.12.1 ZonesThe Proxmox VE firewall groups the network into the following logical zones:HostTraffic from/to a cluster nodeVMTraffic from/to a specific VMFor each zone, you can define firewall rules for incoming and/or outgoing traffic.12.2 Configuration FilesAll firewall related configuration is stored on the proxmox cluster file system. So those files are automaticallydistributed to all cluster nodes, and the pve-firewall service updates the underlying iptables rulesautomatically on changes.You can configure anything using the GUI (i.e. Datacenter → Firewall, or on a Node → Firewall), or youcan edit the configuration files directly using your preferred editor.Firewall configuration files contains sections of key-value pairs. Lines beginning with a # and blank lines areconsidered comments. Sections starts with a header line containing the section name enclosed in [ and ]. Proxmox VE Administration Guide 176 / 32812.2.1 Cluster Wide SetupThe cluster wide firewall configuration is stored at:/etc/pve/firewall/cluster.fwThe configuration can contain the following sections:[OPTIONS]This is used to set cluster wide firewall options.enable: <integer> (0 - N)Enable or disable the firewall cluster wide.policy_in: <ACCEPT | DROP | REJECT>Input policy.policy_out: <ACCEPT | DROP | REJECT>Output policy.[RULES]This sections contains cluster wide firewall rules for all nodes.[IPSET <name>]Cluster wide IP set definitions.[GROUP <name>]Cluster wide security group definitions.[ALIASES]Cluster wide Alias definitions.Enabling the FirewallThe firewall is completely disabled by default, so you need to set the enable option here:[OPTIONS]# enable firewall (cluster wide setting, default is disabled)enable: 1ImportantIf you enable the firewall, traffic to all hosts is blocked by default. Only exceptions is WebGUI(8006)and ssh(22) from your local network.If you want to administrate your Proxmox VE hosts from remote, you need to create rules to allow traffic fromthose remote IPs to the web GUI (port 8006). You may also want to allow ssh (port 22), and maybe SPICE(port 3128). Proxmox VE Administration Guide 6 / 328We also introduced a new REST API, with a complete declarative specification written in JSON-Schema.This enabled other people to integrate Proxmox VE into their infrastructure, and made it easy to provideadditional services.Also, the new REST API made it possible to replace the original user interface with a modern HTML5application using JavaScript. We also replaced the old Java based VNC console code with noVNC. Soyou only need a web browser to manage your VMs.The support for various storage types is another big task. Notably, Proxmox VE was the first distribution toship ZFS on Linux by default in 2014. Another milestone was the ability to run and manage Ceph storage onthe hypervisor nodes. Such setups are extremely cost effective.When we started we were among the first companies providing commercial support for KVM. The KVMproject itself continuously evolved, and is now a widely used hypervisor. New features arrive with eachrelease. We developed the KVM live backup feature, which makes it possible to create snapshot backups onany storage type.The most notable change with version 4.0 was the move from OpenVZ to LXC. Containers are now deeplyintegrated, and they can use the same storage and network features as virtual machines.1.11 Improving the Proxmox VE DocumentationDepending on which issue you want to improve, you can use a variety of communication mediums to reachthe developers.If you notice an error in the current documentation, use the Proxmox bug tracker and propose an alternatetext/wording.If you want to propose new content, it depends on what you want to document:• if the content is specific to your setup, a wiki article is the best option. For instance if you want to documentspecific options for guest systems, like which combination of Qemu drivers work best with a less popularOS, this is a perfect fit for a wiki article.• if you think the content is generic enough to be of interest for all users, then you should try to get it into thereference documentation. The reference documentation is written in the easy to use asciidoc documentformat. Editing the official documentation requires to clone the git repository at git://git.proxmox.com/git/pve-docs.git and then follow the README.adoc document.Improving the documentation is just as easy as editing a Wikipedia article and is an interesting foray in thedevelopment of a large opensource project.NoteIf you are interested in working on the Proxmox VE codebase, the Developer Documentation wiki articlewill show you where to start. Proxmox VE Administration Guide 177 / 328TipPlease open a SSH connection to one of your Proxmox VE hosts before enabling the firewall. That wayyou still have access to the host if something goes wrong .To simplify that task, you can instead create an IPSet called “management”, and add all remote IPs there.This creates all required firewall rules to access the GUI from remote.12.2.2 Host Specific ConfigurationHost related configuration is read from:/etc/pve/nodes/<nodename>/host.fwThis is useful if you want to overwrite rules from cluster.fw config. You can also increase log verbosity,and set netfilter related options. The configuration can contain the following sections:[OPTIONS]This is used to set host related firewall options.enable: <boolean>Enable host firewall rules.log_level_in: <alert | crit | debug | emerg | err | info | nolog |notice | warning>Log level for incoming traffic.log_level_out: <alert | crit | debug | emerg | err | info | nolog |notice | warning>Log level for outgoing traffic.ndp: <boolean>Enable NDP.nf_conntrack_max: <integer> (32768 - N)Maximum number of tracked connections.nf_conntrack_tcp_timeout_established: <integer> (7875 - N)Conntrack established timeout.nosmurfs: <boolean>Enable SMURFS filter.smurf_log_level: <alert | crit | debug | emerg | err | info | nolog| notice | warning>Log level for SMURFS filter. Proxmox VE Administration Guide 178 / 328tcp_flags_log_level: <alert | crit | debug | emerg | err | info |nolog | notice | warning>Log level for illegal tcp flags filter.tcpflags: <boolean>Filter illegal combinations of TCP flags.[RULES]This sections contains host specific firewall rules.12.2.3 VM/Container ConfigurationVM firewall configuration is read from:/etc/pve/firewall/<VMID>.fwand contains the following data:[OPTIONS]This is used to set VM/Container related firewall options.dhcp: <boolean>Enable DHCP.enable: <boolean>Enable/disable firewall rules.ipfilter: <boolean>Enable default IP filters. This is equivalent to adding an empty ipfilter-net<id> ipset for every interface.Such ipsets implicitly contain sane default restrictions such as restricting IPv6 link local addresses tothe one derived from the interface’s MAC address. For containers the configured IP addresses will beimplicitly added.log_level_in: <alert | crit | debug | emerg | err | info | nolog |notice | warning>Log level for incoming traffic.log_level_out: <alert | crit | debug | emerg | err | info | nolog |notice | warning>Log level for outgoing traffic.macfilter: <boolean>Enable/disable MAC address filter.ndp: <boolean>Enable NDP. Proxmox VE Administration Guide 179 / 328policy_in: <ACCEPT | DROP | REJECT>Input policy.policy_out: <ACCEPT | DROP | REJECT>Output policy.radv: <boolean>Allow sending Router Advertisement.[RULES]This sections contains VM/Container firewall rules.[IPSET <name>]IP set definitions.[ALIASES]IP Alias definitions.Enabling the Firewall for VMs and ContainersEach virtual network device has its own firewall enable flag. So you can selectively enable the firewall foreach interface. This is required in addition to the general firewall enable option.The firewall requires a special network device setup, so you need to restart the VM/container after enablingthe firewall on a network interface.12.3 Firewall RulesFirewall rules consists of a direction (IN or OUT) and an action (ACCEPT, DENY, REJECT). You can alsospecify a macro name. Macros contain predefined sets of rules and options. Rules can be disabled byprefixing them with |.Firewall rules syntax[RULES]DIRECTION ACTION [OPTIONS]|DIRECTION ACTION [OPTIONS] # disabled ruleDIRECTION MACRO(ACTION) [OPTIONS] # use predefined macroThe following options can be used to refine rule matches.--dest <string>Restrict packet destination address. This can refer to a single IP address, an IP set (+ipsetname) oran IP alias definition. You can also specify an address range like 20.34.101.207-201.3.9.99, or a listof IP addresses and networks (entries are separated by comma). Please do not mix IPv4 and IPv6addresses inside such lists. Proxmox VE Administration Guide 180 / 328--dport <string>Restrict TCP/UDP destination port. You can use service names or simple numbers (0-65535), asdefined in /etc/services. Port ranges can be specified with \d+:\d+, for example 80:85, and you canuse comma separated list to match several ports or ranges.--iface <string>Network interface name. You have to use network configuration key names for VMs and containers(net\d+). Host related rules can use arbitrary strings.--proto <string>IP protocol. You can use protocol names (tcp/udp) or simple numbers, as defined in /etc/protocols.--source <string>Restrict packet source address. This can refer to a single IP address, an IP set (+ipsetname) or anIP alias definition. You can also specify an address range like 20.34.101.207-201.3.9.99, or a listof IP addresses and networks (entries are separated by comma). Please do not mix IPv4 and IPv6addresses inside such lists.--sport <string>Restrict TCP/UDP source port. You can use service names or simple numbers (0-65535), as definedin /etc/services. Port ranges can be specified with \d+:\d+, for example 80:85, and you can use commaseparated list to match several ports or ranges.Here are some examples:[RULES]IN SSH(ACCEPT) -i net0IN SSH(ACCEPT) -i net0 # a commentIN SSH(ACCEPT) -i net0 -source 192.168.2.192 # only allow SSH from ←-192.168.2.192IN SSH(ACCEPT) -i net0 -source 10.0.0.1-10.0.0.10 # accept SSH for ip rangeIN SSH(ACCEPT) -i net0 -source 10.0.0.1,10.0.0.2,10.0.0.3 #accept ssh for ←-ip listIN SSH(ACCEPT) -i net0 -source +mynetgroup # accept ssh for ipset ←-mynetgroupIN SSH(ACCEPT) -i net0 -source myserveralias #accept ssh for alias ←-myserveralias|IN SSH(ACCEPT) -i net0 # disabled ruleIN DROP # drop all incoming packagesOUT ACCEPT # accept all outgoing packages12.4 Security GroupsA security group is a collection of rules, defined at cluster level, which can be used in all VMs’ rules. Forexample you can define a group named “webserver” with rules to open the http and https ports. Proxmox VE Administration Guide 181 / 328# /etc/pve/firewall/cluster.fw[group webserver]IN ACCEPT -p tcp -dport 80IN ACCEPT -p tcp -dport 443Then, you can add this group to a VM’s firewall# /etc/pve/firewall/<VMID>.fw[RULES]GROUP webserver12.5 IP AliasesIP Aliases allow you to associate IP addresses of networks with a name. You can then refer to those names:• inside IP set definitions• in source and dest properties of firewall rules12.5.1 Standard IP Alias local_networkThis alias is automatically defined. Please use the following command to see assigned values:# pve-firewall localnetlocal hostname: examplelocal IP address: 192.168.2.100network auto detect: 192.168.0.0/20using detected local_network: 192.168.0.0/20The firewall automatically sets up rules to allow everything needed for cluster communication (corosync, API,SSH) using this alias.The user can overwrite these values in the cluster.fw alias section. If you use a single host on a publicnetwork, it is better to explicitly assign the local IP address# /etc/pve/firewall/cluster.fw[ALIASES]local_network 1.2.3.4 # use the single ip address12.6 IP SetsIP sets can be used to define groups of networks and hosts. You can refer to them with ‘+name` in thefirewall rules’ source and dest properties.The following example allows HTTP traffic from the management IP set.IN HTTP(ACCEPT) -source +management Proxmox VE Administration Guide 182 / 32812.6.1 Standard IP set managementThis IP set applies only to host firewalls (not VM firewalls). Those IPs are allowed to do normal managementtasks (PVE GUI, VNC, SPICE, SSH).The local cluster network is automatically added to this IP set (alias cluster_network), to enable inter-host cluster communication. (multicast,ssh,. . . )# /etc/pve/firewall/cluster.fw[IPSET management]192.168.2.10192.168.2.10/2412.6.2 Standard IP set blacklistTraffic from these IPs is dropped by every host’s and VM’s firewall.# /etc/pve/firewall/cluster.fw[IPSET blacklist]77.240.159.182213.87.123.0/2412.6.3 Standard IP set ipfilter-net*These filters belong to a VM’s network interface and are mainly used to prevent IP spoofing. If such a setexists for an interface then any outgoing traffic with a source IP not matching its interface’s correspondingipfilter set will be dropped.For containers with configured IP addresses these sets, if they exist (or are activated via the general IPFilter option in the VM’s firewall’s options tab), implicitly contain the associated IP addresses.For both virtual machines and containers they also implicitly contain the standard MAC-derived IPv6 link-localaddress in order to allow the neighbor discovery protocol to work./etc/pve/firewall/<VMID>.fw[IPSET ipfilter-net0] # only allow specified IPs on net0192.168.2.1012.7 Services and CommandsThe firewall runs two service daemons on each node:• pvefw-logger: NFLOG daemon (ulogd replacement).• pve-firewall: updates iptables rules Proxmox VE Administration Guide 183 / 328There is also a CLI command named pve-firewall, which can be used to start and stop the firewallservice:# pve-firewall start# pve-firewall stopTo get the status use:# pve-firewall statusThe above command reads and compiles all firewall rules, so you will see warnings if your firewall configu-ration contains any errors.If you want to see the generated iptables rules you can use:# iptables-save12.8 Tips and Tricks12.8.1 How to allow FTPFTP is an old style protocol which uses port 21 and several other dynamic ports. So you need a rule toaccept port 21. In addition, you need to load the ip_conntrack_ftp module. So please run:modprobe ip_conntrack_ftpand add ip_conntrack_ftp to /etc/modules (so that it works after a reboot).12.8.2 Suricata IPS integrationIf you want to use the Suricata IPS (Intrusion Prevention System), it’s possible.Packets will be forwarded to the IPS only after the firewall ACCEPTed them.Rejected/Dropped firewall packets don’t go to the IPS.Install suricata on proxmox host:# apt-get install suricata# modprobe nfnetlink_queueDon’t forget to add nfnetlink_queue to /etc/modules for next reboot.Then, enable IPS for a specific VM with:# /etc/pve/firewall/<VMID>.fw[OPTIONS]ips: 1ips_queues: 0ips_queues will bind a specific cpu queue for this VM.Available queues are defined in# /etc/default/suricataNFQUEUE=0 Proxmox VE Administration Guide 184 / 32812.9 Notes on IPv6The firewall contains a few IPv6 specific options. One thing to note is that IPv6 does not use the ARP protocolanymore, and instead uses NDP (Neighbor Discovery Protocol) which works on IP level and thus needs IPaddresses to succeed. For this purpose link-local addresses derived from the interface’s MAC address areused. By default the NDP option is enabled on both host and VM level to allow neighbor discovery (NDP)packets to be sent and received.Beside neighbor discovery NDP is also used for a couple of other things, like autoconfiguration and adver-tising routers.By default VMs are allowed to send out router solicitation messages (to query for a router), and to receiverouter advertisement packets. This allows them to use stateless auto configuration. On the other hand VMscannot advertise themselves as routers unless the “Allow Router Advertisement” (radv: 1) option is set.As for the link local addresses required for NDP, there’s also an “IP Filter” (ipfilter: 1) option whichcan be enabled which has the same effect as adding an ipfilter-net*ipset for each of the VM’snetwork interfaces containing the corresponding link local addresses. (See the Standard IP set ipfilter-net*section for details.)12.10 Ports used by Proxmox VE• Web interface: 8006• VNC Web console: 5900-5999• SPICE proxy: 3128• sshd (used for cluster actions): 22• rpcbind: 111• corosync multicast (if you run a cluster): 5404, 5405 UDP Proxmox VE Administration Guide 185 / 328Chapter 13User ManagementProxmox VE supports multiple authentication sources, e.g. Linux PAM, an integrated Proxmox VE authenti-cation server, LDAP, Microsoft Active Directory.By using the role based user- and permission management for all objects (VMs, storages, nodes, etc.)granular access can be defined.13.1 UsersProxmox VE stores user attributes in /etc/pve/user.cfg. Passwords are not stored here, users areinstead associated with authentication realms described below. Therefore a user is internally often identifiedby its name and realm in the form <userid>@<realm>.Each user entry in this file contains the following information:• First name• Last name• E-mail address• Group memberships• An optional Expiration date• A comment or note about this user• Whether this user is enabled or disabled• Optional two factor authentication keys13.1.1 System administratorThe system’s root user can always log in via the Linux PAM realm and is an unconfined administrator. Thisuser cannot be deleted, but attributes can still be changed and system mails will be sent to the email addressassigned to this user. Proxmox VE Administration Guide 186 / 32813.1.2 GroupsEach user can be member of several groups. Groups are the preferred way to organize access permissions.You should always grant permission to groups instead of using individual users. That way you will get a muchshorter access control list which is easier to handle.13.2 Authentication RealmsAs Proxmox VE users are just counterparts for users existing on some external realm, the realms have to beconfigured in /etc/pve/domains.cfg. The following realms (authentication methods) are available:Linux PAM standard authenticationIn this case a system user has to exist (e.g. created via the adduser command) on all nodes theuser is allowed to login, and the user authenticates with their usual system password.useradd heinzpasswd heinzgroupadd watchmanusermod -a -G watchman heinzProxmox VE authentication serverThis is a unix like password store (/etc/pve/priv/shadow.cfg). Password are encryptedusing the SHA-256 hash method. This is the most convenient method for small (or even medium)installations where users do not need access to anything outside of Proxmox VE. In this case usersare fully managed by Proxmox VE and are able to change their own passwords via the GUI.LDAPIt is possible to authenticate users via an LDAP server (e.g. openldap). The server and an optionalfallback server can be configured and the connection can be encrypted via SSL.Users are searched under a Base Domain Name (base_dn), with the user name found in the attributespecified in the User Attribute Name (user_attr) field.For instance, if a user is represented via the following ldif dataset:# user1 of People at ldap-test.comdn: uid=user1,ou=People,dc=ldap-test,dc=comobjectClass: topobjectClass: personobjectClass: organizationalPersonobjectClass: inetOrgPersonuid: user1cn: Test User 1sn: Testersdescription: This is the first test user.The Base Domain Name would be ou=People,dc=ldap-test,dc=com and the user attributewould be uid.If Proxmox VE needs to authenticate (bind) to the ldap server before being able to query and au-thenticate users, a bind domain name can be configured via the bind_dn property in /etc/pve/ Proxmox VE Administration Guide iiCopyright © 2017 Proxmox Server Solutions GmbhPermission is granted to copy, distribute and/or modify this document under the terms of the GNU FreeDocumentation License, Version 1.3 or any later version published by the Free Software Foundation; withno Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.A copy of the license is included in the section entitled "GNU Free Documentation License". Proxmox VE Administration Guide 7 / 328Chapter 2Installing Proxmox VEProxmox VE is based on Debian and comes with an installation CD-ROM which includes a complete Debiansystem ("stretch" for version 5.x) as well as all necessary Proxmox VE packages.The installer just asks you a few questions, then partitions the local disk(s), installs all required packages,and configures the system including a basic network setup. You can get a fully functional system within afew minutes. This is the preferred and recommended installation method.Alternatively, Proxmox VE can be installed on top of an existing Debian system. This option is only recom-mended for advanced users since detail knowledge about Proxmox VE is necessary.2.1 System RequirementsFor production servers, high quality server equipment is needed. Keep in mind, if you run 10 Virtual Serverson one machine and you then experience a hardware failure, 10 services are lost. Proxmox VE supportsclustering, this means that multiple Proxmox VE installations can be centrally managed thanks to the includedcluster functionality.Proxmox VE can use local storage (DAS), SAN, NAS and also distributed storage (Ceph RBD). For detailssee chapter storage Chapter 8.2.1.1 Minimum Requirements, for Evaluation• CPU: 64bit (Intel EMT64 or AMD64)• Intel VT/AMD-V capable CPU/Mainboard for KVM Full Virtualization support• RAM: 1 GB RAM, plus additional RAM used for guests• Hard drive• One NIC Proxmox VE Administration Guide 187 / 328domains.cfg. Its password then has to be stored in /etc/pve/priv/ldap/<realmname>.pw (e.g. /etc/pve/priv/ldap/my-ldap.pw). This file should contain a single line containingthe raw password.Microsoft Active DirectoryA server and authentication domain need to be specified. Like with ldap an optional fallback server,optional port, and SSL encryption can be configured.13.3 Two factor authenticationEach realm can optionally be secured additionally by two factor authentication. This can be done by selectingone of the available methods via the TFA dropdown box when adding or editing an Authentication Realm.When a realm has TFA enabled it becomes a requirement and only users with configured TFA will be able tologin.Currently there are two methods available:Time based OATH (TOTP)This uses the standard HMAC-SHA1 algorithm where the current time is hashed with the user’s con-figured key. The time step and password length parameters are configured.A user can have multiple keys configured (separated by spaces), and the keys can be specified inBase32 (RFC3548) or hexadecimal notation.Proxmox VE provides a key generation tool (oathkeygen) which prints out a random key in Base32notation which can be used directly with various OTP tools, such as the oathtool command linetool, the Google authenticator or FreeOTP Android apps.YubiKey OTPFor authenticating via a YubiKey a Yubico API ID, API KEY and validation server URL must be con-figured, and users must have a YubiKey available. In order to get the key ID from a YubiKey, youcan trigger the YubiKey once after connecting it to USB and copy the first 12 characters of the typedpassword into the user’s Key IDs field.Please refer to the YubiKey OTP documentation for how to use the YubiCloud or host your own verifi-cation server.13.4 Permission ManagementIn order for a user to perform an action (such as listing, modifying or deleting a parts of a VM configuration),the user needs to have the appropriate permissions.Proxmox VE uses a role and path based permission management system. An entry in the permissions tableallows a user or group to take on a specific role when accessing an object or path. This means an such anaccess rule can be represented as a triple of (path, user, role) or (path, group, role), with the role containinga set of allowed actions, and the path representing the target of these actions. Proxmox VE Administration Guide 188 / 32813.4.1 RolesA role is simply a list of privileges. Proxmox VE comes with a number of predefined roles which satisfiesmost needs.• Administrator: has all privileges• NoAccess: has no privileges (used to forbid access)• PVEAdmin: can do most things, but miss rights to modify system settings (Sys.PowerMgmt, Sys.Modify, Realm.Allocate).• PVEAuditor: read only access• PVEDatastoreAdmin: create and allocate backup space and templates• PVEDatastoreUser: allocate backup space and view storage• PVEPoolAdmin: allocate pools• PVESysAdmin: User ACLs, audit, system console and system logs• PVETemplateUser: view and clone templates• PVEUserAdmin: user administration• PVEVMAdmin: fully administer VMs• PVEVMUser: view, backup, config CDROM, VM console, VM power managementYou can see the whole set of predefined roles on the GUI.Adding new roles can currently only be done from the command line, like this:pveum roleadd PVE_Power-only -privs "VM.PowerMgmt VM.Console"pveum roleadd Sys_Power-only -privs "Sys.PowerMgmt Sys.Console"13.4.2 PrivilegesA privilege is the right to perform a specific action. To simplify management, lists of privileges are groupedinto roles, which can then be used in the permission table. Note that privileges cannot directly be assignedto users and paths without being part of a role.We currently use the following privileges:Node / System related privileges• Permissions.Modify: modify access permissions• Sys.PowerMgmt: Node power management (start, stop, reset, shutdown, . . . )• Sys.Console: console access to Node• Sys.Syslog: view Syslog• Sys.Audit: view node status/config, Corosync cluster config and HA config Proxmox VE Administration Guide 189 / 328• Sys.Modify: create/remove/modify node network parameters• Group.Allocate: create/remove/modify groups• Pool.Allocate: create/remove/modify a pool• Realm.Allocate: create/remove/modify authentication realms• Realm.AllocateUser: assign user to a realm• User.Modify: create/remove/modify user access and details.Virtual machine related privileges• VM.Allocate: create/remove new VM to server inventory• VM.Migrate: migrate VM to alternate server on cluster• VM.PowerMgmt: power management (start, stop, reset, shutdown, . . . )• VM.Console: console access to VM• VM.Monitor: access to VM monitor (kvm)• VM.Backup: backup/restore VMs• VM.Audit: view VM config• VM.Clone: clone/copy a VM• VM.Config.Disk: add/modify/delete Disks• VM.Config.CDROM: eject/change CDROM• VM.Config.CPU: modify CPU settings• VM.Config.Memory: modify Memory settings• VM.Config.Network: add/modify/delete Network devices• VM.Config.HWType: modify emulated HW type• VM.Config.Options: modify any other VM configuration• VM.Snapshot: create/remove VM snapshotsStorage related privileges• Datastore.Allocate: create/remove/modify a data store, delete volumes• Datastore.AllocateSpace: allocate space on a datastore• Datastore.AllocateTemplate: allocate/upload templates and iso images• Datastore.Audit: view/browse a datastore13.4.3 Objects and PathsAccess permissions are assigned to objects, such as a virtual machines, storages or pools of resources.We use file system like paths to address these objects. These paths form a natural tree, and permissions ofhigher levels (shorter path) can optionally be propagated down within this hierarchy.Paths can be templated. When an API call requires permissions on a templated path, the path may containreferences to parameters of the API call. These references are specified in curly braces. Some parametersare implicitly taken from the API call’s URI. For instance the permission path /nodes/{node} when Proxmox VE Administration Guide 190 / 328calling /nodes/mynode/status requires permissions on /nodes/mynode, while the path {path} in aPUT request to /access/acl refers to the method’s path parameter.Some examples are:• /nodes/{node}: Access to Proxmox VE server machines• /vms: Covers all VMs• /vms/{vmid}: Access to specific VMs• /storage/{storeid}: Access to a storages• /pool/{poolname}: Access to VMs part of a pool• /access/groups: Group administration• /access/realms/{realmid}: Administrative access to realmsInheritanceAs mentioned earlier, object paths form a file system like tree, and permissions can be inherited down thattree (the propagate flag is set by default). We use the following inheritance rules:• Permissions for individual users always replace group permissions.• Permissions for groups apply when the user is member of that group.• Permissions replace the ones inherited from an upper level.13.4.4 PoolsPools can be used to group a set of virtual machines and data stores. You can then simply set permissionson pools (/pool/{poolid}), which are inherited to all pool members. This is a great way simplify accesscontrol.13.4.5 What permission do I need?The required API permissions are documented for each individual method, and can be found at http://pve.proxmox.com/-pve-docs/api-viewer/The permissions are specified as a list which can be interpreted as a tree of logic and access-check func-tions:["and", <subtests>...] and ["or", <subtests>...]Each(and) or any(or) further element in the current list has to be true.["perm", <path>, [ <privileges>... ], <options>...]The path is a templated parameter (see Objects and Paths). All (or , if the any option is used,any) of the listed privileges must be allowed on the specified path. If a require-param option isspecified, then its specified parameter is required even if the API call’s schema otherwise lists it asbeing optional. Proxmox VE Administration Guide 191 / 328["userid-group", [ <privileges>... ], <options>...]The caller must have any of the listed privileges on /access/groups. In addition there are twopossible checks depending on whether the groups_param option is set:• groups_param is set: The API call has a non-optional groups parameter and the caller musthave any of the listed privileges on all of the listed groups.• groups_param is not set: The user passed via the userid parameter must exist and be part ofa group on which the caller has any of the listed privileges (via the /access/groups/<group>path).["userid-param", "self"]The value provided for the API call’s userid parameter must refer to the user performing the action.(Usually in conjunction with or, to allow users to perform an action on themselves even if they don’thave elevated privileges.)["userid-param", "Realm.AllocateUser"]The user needs Realm.AllocateUser access to /access/realm/<realm>, with <realm> referring to the realm of the user passed via the userid parameter. Note that the user doesnot need to exist in order to be associated with a realm, since user IDs are passed in the form of<username>@<realm>.["perm-modify", <path>]The path is a templated parameter (see Objects and Paths). The user needs either the Permissions.Modify privilege, or, depending on the path, the following privileges as a possible substitute:• /storage/...: additionally requires ’Datastore.Allocate`• /vms/...: additionally requires ’VM.Allocate`• /pool/...: additionally requires ’Pool.Allocate`If the path is empty, Permission.Modify on /access is required.13.5 Command Line ToolMost users will simply use the GUI to manage users. But there is also a full featured command line toolcalled pveum (short for “Proxmox VE User Manager”). Please note that all Proxmox VE command line toolsare wrappers around the API, so you can also access those function through the REST API.Here are some simple usage examples. To show help type:pveumor (to show detailed help about a specific command)pveum help useraddCreate a new user:pveum useradd testuser@pve -comment "Just a test" Proxmox VE Administration Guide 192 / 328Set or Change the password (not all realms support that):pveum passwd testuser@pveDisable a user:pveum usermod testuser@pve -enable 0Create a new group:pveum groupadd testgroupCreate a new role:pveum roleadd PVE_Power-only -privs "VM.PowerMgmt VM.Console"13.6 Real World Examples13.6.1 Administrator GroupOne of the most wanted features was the ability to define a group of users with full administrator rights(without using the root account).Define the group:pveum groupadd admin -comment "System Administrators"Then add the permission:pveum aclmod / -group admin -role AdministratorYou can finally add users to the new admin group:pveum usermod testuser@pve -group admin13.6.2 AuditorsYou can give read only access to users by assigning the PVEAuditor role to users or groups.Example1: Allow user joe@pve to see everythingpveum aclmod / -user joe@pve -role PVEAuditorExample1: Allow user joe@pve to see all virtual machinespveum aclmod /vms -user joe@pve -role PVEAuditor Proxmox VE Administration Guide 193 / 32813.6.3 Delegate User ManagementIf you want to delegate user management to user joe@pve you can do that with:pveum aclmod /access -user joe@pve -role PVEUserAdminUser joe@pve can now add and remove users, change passwords and other user attributes. This is a verypowerful role, and you most likely want to limit that to selected realms and groups. The following exampleallows joe@pve to modify users within realm pve if they are members of group customers:pveum aclmod /access/realm/pve -user joe@pve -role PVEUserAdminpveum aclmod /access/groups/customers -user joe@pve -role PVEUserAdminNoteThe user is able to add other users, but only if they are members of group customers and within realmpve.13.6.4 PoolsAn enterprise is usually structured into several smaller departments, and it is common that you want toassign resources to them and delegate management tasks. A pool is simply a set of virtual machines anddata stores. You can create pools on the GUI. After that you can add resources to the pool (VMs, Storage).You can also assign permissions to the pool. Those permissions are inherited to all pool members.Lets assume you have a software development department, so we first create a grouppveum groupadd developers -comment "Our software developers"Now we create a new user which is a member of that grouppveum useradd developer1@pve -group developers -passwordNoteThe -password parameter will prompt you for a passwordI assume we already created a pool called “dev-pool” on the GUI. So we can now assign permission to thatpool:pveum aclmod /pool/dev-pool/ -group developers -role PVEAdminOur software developers can now administrate the resources assigned to that pool. Proxmox VE Administration Guide 194 / 328Chapter 14High AvailabilityOur modern society depends heavily on information provided by computers over the network. Mobile devicesamplified that dependency, because people can access the network any time from anywhere. If you providesuch services, it is very important that they are available most of the time.We can mathematically define the availability as the ratio of (A) the total time a service is capable of beingused during a given interval to (B) the length of the interval. It is normally expressed as a percentage ofuptime in a given year.Table 14.1: Availability - Downtime per YearAvailability % Downtime per year99 3.65 days99.9 8.76 hours99.99 52.56 minutes99.999 5.26 minutes99.9999 31.5 seconds99.99999 3.15 secondsThere are several ways to increase availability. The most elegant solution is to rewrite your software, so thatyou can run it on several host at the same time. The software itself need to have a way to detect errors and dofailover. This is relatively easy if you just want to serve read-only web pages. But in general this is complex,and sometimes impossible because you cannot modify the software yourself. The following solutions workswithout modifying the software:• Use reliable “server” componentsNoteComputer components with same functionality can have varying reliability numbers, depending on thecomponent quality. Most vendors sell components with higher reliability as “server” components - usuallyat higher price.• Eliminate single point of failure (redundant components) Proxmox VE Administration Guide 195 / 328– use an uninterruptible power supply (UPS)– use redundant power supplies on the main boards– use ECC-RAM– use redundant network hardware– use RAID for local storage– use distributed, redundant storage for VM data• Reduce downtime– rapidly accessible administrators (24/7)– availability of spare parts (other nodes in a Proxmox VE cluster)– automatic error detection (provided by ha-manager)– automatic failover (provided by ha-manager)Virtualization environments like Proxmox VE make it much easier to reach high availability because theyremove the “hardware” dependency. They also support to setup and use redundant storage and networkdevices. So if one host fail, you can simply start those services on another host within your cluster.Even better, Proxmox VE provides a software stack called ha-manager, which can do that automaticallyfor you. It is able to automatically detect errors and do automatic failover.Proxmox VE ha-manager works like an “automated” administrator. First, you configure what resources(VMs, containers, . . . ) it should manage. ha-manager then observes correct functionality, and handlesservice failover to another node in case of errors. ha-manager can also handle normal user requestswhich may start, stop, relocate and migrate a service.But high availability comes at a price. High quality components are more expensive, and making themredundant duplicates the costs at least. Additional spare parts increase costs further. So you should carefullycalculate the benefits, and compare with those additional costs.TipIncreasing availability from 99% to 99.9% is relatively simply. But increasing availability from 99.9999% to99.99999% is very hard and costly. ha-manager has typical error detection and failover times of about2 minutes, so you can get no more than 99.999% availability.14.1 RequirementsYou must meet the following requirements before you start with HA:• at least three cluster nodes (to get reliable quorum)• shared storage for VMs and containers• hardware redundancy (everywhere)• use reliable “server” components• hardware watchdog - if not available we fall back to the linux kernel software watchdog (softdog)• optional hardware fencing devices Proxmox VE Administration Guide 196 / 32814.2 ResourcesWe call the primary management unit handled by ha-manager a resource. A resource (also called “ser-vice”) is uniquely identified by a service ID (SID), which consists of the resource type and an type specificID, e.g.: vm:100. That example would be a resource of type vm (virtual machine) with the ID 100.For now we have two important resources types - virtual machines and containers. One basic idea hereis that we can bundle related software into such VM or container, so there is no need to compose one bigservice from other services, like it was done with rgmanager. In general, a HA managed resource shouldnot depend on other resources.14.3 Management TasksThis section provides a short overview of common management tasks. The first step is to enable HA for aresource. This is done by adding the resource to the HA resource configuration. You can do this using theGUI, or simply use the command line tool, for example:# ha-manager add vm:100The HA stack now tries to start the resources and keeps it running. Please note that you can configure the“requested” resources state. For example you may want that the HA stack stops the resource:# ha-manager set vm:100 --state stoppedand start it again later:# ha-manager set vm:100 --state startedYou can also use the normal VM and container management commands. They automatically forward thecommands to the HA stack, so# qm start 100simply sets the requested state to started. Same applied to qm stop, which sets the requested stateto stopped.NoteThe HA stack works fully asynchronous and needs to communicate with other cluster members. So ittakes some seconds until you see the result of such actions.To view the current HA resource configuration use:# ha-manager configvm:100state stoppedAnd you can view the actual HA manager and resource state with:# ha-manager statusquorum OKmaster node1 (active, Wed Nov 23 11:07:23 2016)lrm elsa (active, Wed Nov 23 11:07:19 2016)service vm:100 (node1, started) Proxmox VE Administration Guide 8 / 3282.1.2 Recommended System Requirements• CPU: 64bit (Intel EMT64 or AMD64), Multi core CPU recommended• Intel VT/AMD-V capable CPU/Mainboard for KVM Full Virtualization support• RAM: 8 GB RAM, plus additional RAM used for guests• Hardware RAID with batteries protected write cache (“BBU”) or flash based protection• Fast hard drives, best results with 15k rpm SAS, Raid10• At least two NICs, depending on the used storage technology you need more2.1.3 Simple Performance OverviewOn an installed Proxmox VE system, you can run the included pveperf script to obtain an overview of theCPU and hard disk performance.Notethis is just a very quick and general benchmark. More detailed tests are recommended, especially regard-ing the I/O performance of your system.2.1.4 Supported web browsers for accessing the web interfaceTo use the web interface you need a modern browser, this includes:• Firefox, a release from the current year, or the latest Extended Support Release• Chrome, a release from the current year• the Microsoft currently supported versions of Internet Explorer (as of 2016, this means IE 11 or IE Edge)• the Apple currently supported versions of Safari (as of 2016, this means Safari 9)If Proxmox VE detects you’re connecting from a mobile device, you will be redirected to a lightweight touch-based UI.2.2 Using the Proxmox VE Installation CD-ROMYou can download the ISO from http://www.proxmox.com. It includes the following:• Complete operating system (Debian Linux, 64-bit)• The Proxmox VE installer, which partitions the hard drive(s) with ext4, ext3, xfs or ZFS and installs theoperating system.• Proxmox VE kernel (Linux) with LXC and KVM support Proxmox VE Administration Guide 197 / 328You can also initiate resource migration to other nodes:# ha-manager migrate vm:100 node2This uses online migration and tries to keep the VM running. Online migration needs to transfer all usedmemory over the network, so it is sometimes faster to stop VM, then restart it on the new node. This can bedone using the relocate command:# ha-manager relocate vm:100 node2Finally, you can remove the resource from the HA configuration using the following command:# ha-manager remove vm:100NoteThis does not start or stop the resource.But all HA related task can be done on the GUI, so there is no need to use the command line at all.14.4 How It WorksThis section provides a detailed description of the Proxmox VE HA manager internals. It describes allinvolved daemons and how they work together. To provide HA, two daemons run on each node:pve-ha-lrmThe local resource manager (LRM), which controls the services running on the local node. It readsthe requested states for its services from the current manager status file and executes the respectivecommands.pve-ha-crmThe cluster resource manager (CRM), which makes the cluster wide decisions. It sends commandsto the LRM, processes the results, and moves resources to other nodes if something fails. The CRMalso handles node fencing.NoteLocks are provided by our distributed configuration file system (pmxcfs). They are used to guarantee thateach LRM is active once and working. As a LRM only executes actions when it holds its lock, we canmark a failed node as fenced if we can acquire its lock. This lets us then recover any failed HA servicessecurely without any interference from the now unknown failed node. This all gets supervised by the CRMwhich holds currently the manager master lock. Proxmox VE Administration Guide 198 / 32814.4.1 Service StatesThe CRM use a service state enumeration to record the current service state. We display this state on theGUI and you can query it using the ha-manager command line tool:# ha-manager statusquorum OKmaster elsa (active, Mon Nov 21 07:23:29 2016)lrm elsa (active, Mon Nov 21 07:23:22 2016)service ct:100 (elsa, stopped)service ct:102 (elsa, started)service vm:501 (elsa, started)Here is the list of possible states:stoppedService is stopped (confirmed by LRM). If the LRM detects a stopped service is still running, it will stopit again.request_stopService should be stopped. The CRM waits for confirmation from the LRM.stoppingPending stop request. But the CRM did not get the request so far.startedService is active an LRM should start it ASAP if not already running. If the Service fails and is detectedto be not running the LRM restarts it (see Start Failure Policy Section 14.7).startingPending start request. But the CRM has not got any confirmation from the LRM that the service isrunning.fenceWait for node fencing (service node is not inside quorate cluster partition). As soon as node getsfenced successfully the service will be recovered to another node, if possible (see Fencing Sec-tion 14.6).freezeDo not touch the service state. We use this state while we reboot a node, or when we restart the LRMdaemon (see Package Updates Section 14.9).ignoredAct as if the service were not managed by HA at all. Useful, when full control over the service isdesired temporarily, without removing it from the HA configuration.migrateMigrate service (live) to other node. Proxmox VE Administration Guide 199 / 328errorService is disabled because of LRM errors. Needs manual intervention (see Error Recovery Sec-tion 14.8).queuedService is newly added, and the CRM has not seen it so far.disabledService is stopped and marked as disabled14.4.2 Local Resource ManagerThe local resource manager (pve-ha-lrm) is started as a daemon on boot and waits until the HA clusteris quorate and thus cluster wide locks are working.It can be in three states:wait for agent lockThe LRM waits for our exclusive lock. This is also used as idle state if no service is configured.activeThe LRM holds its exclusive lock and has services configured.lost agent lockThe LRM lost its lock, this means a failure happened and quorum was lost.After the LRM gets in the active state it reads the manager status file in /etc/pve/ha/manager_status and determines the commands it has to execute for the services it owns. For each command aworker gets started, this workers are running in parallel and are limited to at most 4 by default. This defaultsetting may be changed through the datacenter configuration key max_worker. When finished the workerprocess gets collected and its result saved for the CRM.NoteThe default value of at most 4 concurrent workers may be unsuited for a specific setup. For example may4 live migrations happen at the same time, which can lead to network congestions with slower networksand/or big (memory wise) services. Ensure that also in the worst case no congestion happens and lowerthe max_worker value if needed. In the contrary, if you have a particularly powerful high end setup youmay also want to increase it.Each command requested by the CRM is uniquely identifiable by an UID, when the worker finished its resultwill be processed and written in the LRM status file /etc/pve/nodes/<nodename>/lrm_status.There the CRM may collect it and let its state machine - respective the commands output - act on it.The actions on each service between CRM and LRM are normally always synced. This means that the CRMrequests a state uniquely marked by an UID, the LRM then executes this action one time and writes backthe result, also identifiable by the same UID. This is needed so that the LRM does not executes an outdatedcommand. With the exception of the stop and the error command, those two do not depend on the resultproduced and are executed always in the case of the stopped state and once in the case of the error state. Proxmox VE Administration Guide 200 / 328NoteThe HA Stack logs every action it makes. This helps to understand what and also why something happensin the cluster. Here its important to see what both daemons, the LRM and the CRM, did. You may usejournalctl -u pve-ha-lrm on the node(s) where the service is and the same command for thepve-ha-crm on the node which is the current master.14.4.3 Cluster Resource ManagerThe cluster resource manager (pve-ha-crm) starts on each node and waits there for the manager lock,which can only be held by one node at a time. The node which successfully acquires the manager lock getspromoted to the CRM master.It can be in three states:wait for agent lockThe CRM waits for our exclusive lock. This is also used as idle state if no service is configuredactiveThe CRM holds its exclusive lock and has services configuredlost agent lockThe CRM lost its lock, this means a failure happened and quorum was lost.It main task is to manage the services which are configured to be highly available and try to always enforcethe requested state. For example, a service with the requested state started will be started if its not alreadyrunning. If it crashes it will be automatically started again. Thus the CRM dictates the actions which the LRMneeds to execute.When an node leaves the cluster quorum, its state changes to unknown. If the current CRM then can securethe failed nodes lock, the services will be stolen and restarted on another node.When a cluster member determines that it is no longer in the cluster quorum, the LRM waits for a newquorum to form. As long as there is no quorum the node cannot reset the watchdog. This will trigger areboot after the watchdog then times out, this happens after 60 seconds.14.5 ConfigurationThe HA stack is well integrated into the Proxmox VE API. So, for example, HA can be configured via the ha-manager command line interface, or the Proxmox VE web interface - both interfaces provide an easy wayto manage HA. Automation tools can use the API directly.All HA configuration files are within /etc/pve/ha/, so they get automatically distributed to the clusternodes, and all nodes share the same HA configuration. Proxmox VE Administration Guide 201 / 32814.5.1 ResourcesThe resource configuration file /etc/pve/ha/resources.cfg stores the list of resources managedby ha-manager. A resource configuration inside that list look like this:<type>: <name><property> <value>...It starts with a resource type followed by a resource specific name, separated with colon. Together this formsthe HA resource ID, which is used by all ha-manager commands to uniquely identify a resource (example:vm:100 or ct:101). The next lines contain additional properties:comment: <string>Description.group: <string>The HA group identifier.max_relocate: <integer> (0 - N) (default = 1)Maximal number of service relocate tries when a service failes to start.max_restart: <integer> (0 - N) (default = 1)Maximal number of tries to restart the service on a node after its start failed.state: <disabled | enabled | ignored | started | stopped> (default =started)Requested resource state. The CRM reads this state and acts accordingly. Please note that enabledis just an alias for started. Proxmox VE Administration Guide 202 / 328startedThe CRM tries to start the resource. Service state is set to started after successful start. Onnode failures, or when start fails, it tries to recover the resource. If everything fails, service stateit set to error.stoppedThe CRM tries to keep the resource in stopped state, but it still tries to relocate the resourceson node failures.disabledThe CRM tries to put the resource in stopped state, but does not try to relocate the resourceson node failures. The main purpose of this state is error recovery, because it is the only way tomove a resource out of the error state.ignoredThe resource gets removed from the manager status and so the CRM and the LRM do not touchthe resource anymore. All Proxmox VE API calls affecting this resource will be executed, directlybypassing the HA stack. CRM commands will be thrown away while there source is in this state.The resource will not get relocated on node failures.Here is a real world example with one VM and one container. As you see, the syntax of those files is reallysimple, so it is even possible to read or edit those files using your favorite editor:Configuration Example (/etc/pve/ha/resources.cfg)vm: 501state startedmax_relocate 2ct: 102# Note: use default settings for everythingAbove config was generated using the ha-manager command line tool:# ha-manager add vm:501 --state started --max_relocate 2# ha-manager add ct:102 Proxmox VE Administration Guide 203 / 32814.5.2 GroupsThe HA group configuration file /etc/pve/ha/groups.cfg is used to define groups of cluster nodes.A resource can be restricted to run only on the members of such group. A group configuration look like this:group: <group>nodes <node_list><property> <value>...comment: <string>Description.nodes: <node>[:<pri>]{,<node>[:<pri>]}*List of cluster node members, where a priority can be given to each node. A resource bound to agroup will run on the available nodes with the highest priority. If there are more nodes in the highestpriority class, the services will get distributed to those nodes. The priorities have a relative meaningonly.nofailback: <boolean> (default = 0)The CRM tries to run services on the node with the highest priority. If a node with higher priority comesonline, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.restricted: <boolean> (default = 0)Resources bound to restricted groups may only run on nodes defined by the group. The resource willbe placed in the stopped state if no group node member is online. Resources on unrestricted groupsmay run on any cluster node if all group members are offline, but they will migrate back as soon asa group member comes online. One can implement a preferred node behavior using an unrestrictedgroup with only one member. Proxmox VE Administration Guide 204 / 328A common requirement is that a resource should run on a specific node. Usually the resource is able to runon other nodes, so you can define an unrestricted group with a single member:# ha-manager groupadd prefer_node1 --nodes node1For bigger clusters, it makes sense to define a more detailed failover behavior. For example, you may wantto run a set of services on node1 if possible. If node1 is not available, you want to run them equally spliton node2 and node3. If those nodes also fail the services should run on node4. To achieve this youcould set the node list to:# ha-manager groupadd mygroup1 -nodes "node1:2,node2:1,node3:1,node4"Another use case is if a resource uses other resources only available on specific nodes, lets say node1and node2. We need to make sure that HA manager does not use other nodes, so we need to create arestricted group with said nodes:# ha-manager groupadd mygroup2 -nodes "node1,node2" -restrictedAbove commands created the following group configuration fils:Configuration Example (/etc/pve/ha/groups.cfg)group: prefer_node1nodes node1group: mygroup1nodes node2:1,node4,node1:2,node3:1group: mygroup2nodes node2,node1restricted 1The nofailback options is mostly useful to avoid unwanted resource movements during administrationtasks. For example, if you need to migrate a service to a node which hasn’t the highest priority in the group,you need to tell the HA manager to not move this service instantly back by setting the nofailback option. Proxmox VE Administration Guide 205 / 328Another scenario is when a service was fenced and it got recovered to another node. The admin tries torepair the fenced node and brings it up online again to investigate the failure cause and check if it runs stableagain. Setting the nofailback flag prevents that the recovered services move straight back to the fencednode.14.6 FencingOn node failures, fencing ensures that the erroneous node is guaranteed to be offline. This is required tomake sure that no resource runs twice when it gets recovered on another node. This is a really importanttask, because without, it would not be possible to recover a resource on another node.If a node would not get fenced, it would be in an unknown state where it may have still access to sharedresources. This is really dangerous! Imagine that every network but the storage one broke. Now, while notreachable from the public network, the VM still runs and writes to the shared storage.If we then simply start up this VM on another node, we would get a dangerous race conditions becausewe write from both nodes. Such condition can destroy all VM data and the whole VM could be renderedunusable. The recovery could also fail if the storage protects from multiple mounts.14.6.1 How Proxmox VE FencesThere are different methods to fence a node, for example, fence devices which cut off the power from thenode or disable their communication completely. Those are often quite expensive and bring additional criticalcomponents into a system, because if they fail you cannot recover any service.We thus wanted to integrate a simpler fencing method, which does not require additional external hardware.This can be done using watchdog timers.POSSIBLE FENCING METHODS• external power switches• isolate nodes by disabling complete network traffic on the switch• self fencing using watchdog timersWatchdog timers are widely used in critical and dependable systems since the beginning of micro controllers.They are often independent and simple integrated circuits which are used to detect and recover from com-puter malfunctions.During normal operation, ha-manager regularly resets the watchdog timer to prevent it from elapsing. If,due to a hardware fault or program error, the computer fails to reset the watchdog, the timer will elapse andtriggers a reset of the whole server (reboot).Recent server motherboards often include such hardware watchdogs, but these need to be configured. Ifno watchdog is available or configured, we fall back to the Linux Kernel softdog. While still reliable, it is notindependent of the servers hardware, and thus has a lower reliability than a hardware watchdog. Proxmox VE Administration Guide 206 / 32814.6.2 Configure Hardware WatchdogBy default, all hardware watchdog modules are blocked for security reasons. They are like a loaded gunif not correctly initialized. To enable a hardware watchdog, you need to specify the module to load in/etc/default/pve-ha-manager, for example:# select watchdog module (default is softdog)WATCHDOG_MODULE=iTCO_wdtThis configuration is read by the watchdog-mux service, which load the specified module at startup.14.6.3 Recover Fenced ServicesAfter a node failed and its fencing was successful, the CRM tries to move services from the failed node tonodes which are still online.The selection of nodes, on which those services gets recovered, is influenced by the resource groupsettings, the list of currently active nodes, and their respective active service count.The CRM first builds a set out of the intersection between user selected nodes (from group setting) andavailable nodes. It then choose the subset of nodes with the highest priority, and finally select the node withthe lowest active service count. This minimizes the possibility of an overloaded node.CautionOn node failure, the CRM distributes services to the remaining nodes. This increase the servicecount on those nodes, and can lead to high load, especially on small clusters. Please design yourcluster so that it can handle such worst case scenarios.14.7 Start Failure PolicyThe start failure policy comes in effect if a service failed to start on a node once ore more times. It canbe used to configure how often a restart should be triggered on the same node and how often a serviceshould be relocated so that it gets a try to be started on another node. The aim of this policy is to circumventtemporary unavailability of shared resources on a specific node. For example, if a shared storage isn’tavailable on a quorate node anymore, e.g. network problems, but still on other nodes, the relocate policyallows then that the service gets started nonetheless.There are two service start recover policy settings which can be configured specific for each resource.max_restartMaximum number of tries to restart an failed service on the actual node. The default is set to one.max_relocateMaximum number of tries to relocate the service to a different node. A relocate only happens after themax_restart value is exceeded on the actual node. The default is set to one.NoteThe relocate count state will only reset to zero when the service had at least one successful start. Thatmeans if a service is re-started without fixing the error only the restart policy gets repeated. Proxmox VE Administration Guide 9 / 328• Complete toolset for administering virtual machines, containers and all necessary resources• Web based management interface for using the toolsetNoteBy default, the complete server is used and all existing data is removed.Please insert the installation CD-ROM, then boot from that drive. Immediately afterwards you can choosethe following menu options:Install Proxmox VEStart normal installation.Install Proxmox VE (Debug mode)Start installation in debug mode. It opens a shell console at several installation steps, so that youcan debug things if something goes wrong. Please press CTRL-D to exit those debug consoles andcontinue installation. This option is mostly for developers and not meant for general use.Rescue BootThis option allows you to boot an existing installation. It searches all attached hard disks and, if it findsan existing installation, boots directly into that disk using the existing Linux kernel. This can be usefulif there are problems with the boot block (grub), or the BIOS is unable to read the boot block from thedisk.Test MemoryRuns memtest86+. This is useful to check if your memory is functional and error free. Proxmox VE Administration Guide 207 / 32814.8 Error RecoveryIf after all tries the service state could not be recovered it gets placed in an error state. In this state theservice won’t get touched by the HA stack anymore. The only way out is disabling a service:# ha-manager set vm:100 --state disabledThis can also be done in the web interface.To recover from the error state you should do the following:• bring the resource back into a safe and consistent state (e.g.: kill its process if the service could not bestopped)• disable the resource to remove the error flag• fix the error which led to this failures• after you fixed all errors you may request that the service starts again14.9 Package UpdatesWhen updating the ha-manager you should do one node after the other, never all at once for various reasons.First, while we test our software thoughtfully, a bug affecting your specific setup cannot totally be ruled out.Upgrading one node after the other and checking the functionality of each node after finishing the updatehelps to recover from an eventual problems, while updating all could render you in a broken cluster state andis generally not good practice.Also, the Proxmox VE HA stack uses a request acknowledge protocol to perform actions between the clusterand the local resource manager. For restarting, the LRM makes a request to the CRM to freeze all itsservices. This prevents that they get touched by the Cluster during the short time the LRM is restarting.After that the LRM may safely close the watchdog during a restart. Such a restart happens normally during apackage update and, as already stated, an active master CRM is needed to acknowledge the requests fromthe LRM. If this is not the case the update process can take too long which, in the worst case, may result ina reset triggered by the watchdog.14.10 Node MaintenanceIt is sometimes possible to shutdown or reboot a node to do maintenance tasks. Either to replace hardware,or simply to install a new kernel image.14.10.1 ShutdownA shutdown (poweroff ) is usually done if the node is planned to stay down for some time. The LRM stops allmanaged services in that case. This means that other nodes will take over those service afterwards.NoteRecent hardware has large amounts of RAM. So we stop all resources, then restart them to avoid onlinemigration of all that RAM. If you want to use online migration, you need to invoke that manually before youshutdown the node. Proxmox VE Administration Guide 208 / 32814.10.2 RebootNode reboots are initiated with the reboot command. This is usually done after installing a new kernel.Please note that this is different from “shutdown”, because the node immediately starts again.The LRM tells the CRM that it wants to restart, and waits until the CRM puts all resources into the freezestate (same mechanism is used for Package Updates Section 14.9). This prevents that those resources aremoved to other nodes. Instead, the CRM start the resources after the reboot on the same node.14.10.3 Manual Resource MovementLast but not least, you can also move resources manually to other nodes before you shutdown or restart anode. The advantage is that you have full control, and you can decide if you want to use online migration ornot.NotePlease do not kill services like pve-ha-crm, pve-ha-lrm or watchdog-mux. They manage anduse the watchdog, so this can result in a node reboot. Proxmox VE Administration Guide 209 / 328Chapter 15Backup and RestoreBackups are a requirements for any sensible IT deployment, and Proxmox VE provides a fully integratedsolution, using the capabilities of each storage and each guest system type. This allows the system admin-istrator to fine tune via the mode option between consistency of the backups and downtime of the guestsystem.Proxmox VE backups are always full backups - containing the VM/CT configuration and all data. Backupscan be started via the GUI or via the vzdump command line tool.Backup StorageBefore a backup can run, a backup storage must be defined. Refer to the Storage documentation on how toadd a storage. A backup storage must be a file level storage, as backups are stored as regular files. In mostsituations, using a NFS server is a good way to store backups. You can save those backups later to a tapedrive, for off-site archiving.Scheduled BackupBackup jobs can be scheduled so that they are executed automatically on specific days and times, forselectable nodes and guest systems. Configuration of scheduled backups is done at the Datacenter level inthe GUI, which will generate a cron entry in /etc/cron.d/vzdump.15.1 Backup modesThere are several ways to provide consistency (option mode), depending on the guest type.BACKUP MODES FOR VMS:stop modeThis mode provides the highest consistency of the backup, at the cost of a short downtime in the VMoperation. It works by executing an orderly shutdown of the VM, and then runs a background Qemuprocess to backup the VM data. After the backup is started, the VM goes to full operation mode if itwas previously running. Consistency is guaranteed by using the live backup feature. Proxmox VE Administration Guide 210 / 328suspend modeThis mode is provided for compatibility reason, and suspends the VM before calling the snapshotmode. Since suspending the VM results in a longer downtime and does not necessarily improve thedata consistency, the use of the snapshot mode is recommended instead.snapshot modeThis mode provides the lowest operation downtime, at the cost of a small inconstancy risk. It worksby performing a Proxmox VE live backup, in which data blocks are copied while the VM is running. Ifthe guest agent is enabled (agent: 1) and running, it calls guest-fsfreeze-freeze andguest-fsfreeze-thaw to improve consistency.A technical overview of the Proxmox VE live backup for QemuServer can be found online here.NoteProxmox VE live backup provides snapshot-like semantics on any storage type. It does not require thatthe underlying storage supports snapshots. Also please note that since the backups are done via abackground Qemu process, a stopped VM will appear as running for a short amount of time while the VMdisks are being read by Qemu. However the VM itself is not booted, only its disk(s) are read.BACKUP MODES FOR CONTAINERS:stop modeStop the container for the duration of the backup. This potentially results in a very long downtime.suspend modeThis mode uses rsync to copy the container data to a temporary location (see option --tmpdir).Then the container is suspended and a second rsync copies changed files. After that, the containeris started (resumed) again. This results in minimal downtime, but needs additional space to hold thecontainer copy.When the container is on a local file system and the target storage of the backup is an NFS/CIFSserver, you should set --tmpdir to reside on a local file system too, as this will result in a manyfold performance improvement. Use of a local tmpdir is also required if you want to backup a localcontainer using ACLs in suspend mode if the backup storage is an NFS server.snapshot modeThis mode uses the snapshotting facilities of the underlying storage. First, the container will be sus-pended to ensure data consistency. A temporary snapshot of the container’s volumes will be madeand the snapshot content will be archived in a tar file. Finally, the temporary snapshot is deleted again.Notesnapshot mode requires that all backed up volumes are on a storage that supports snapshots. Usingthe backup=no mount point option individual volumes can be excluded from the backup (and thus thisrequirement).NoteBy default additional mount points besides the Root Disk mount point are not included in backups. Forvolume mount points you can set the Backup option to include the mount point in the backup. Device andbind mounts are never backed up as their content is managed outside the Proxmox VE storage library. Proxmox VE Administration Guide 211 / 32815.2 Backup File NamesNewer versions of vzdump encode the guest type and the backup time into the filename, for examplevzdump-lxc-105-2009_10_09-11_04_43.tarThat way it is possible to store several backup in the same directory. The parameter maxfiles can beused to specify the maximum number of backups to keep.15.3 RestoreA backup archive can be restored through the Proxmox VE web GUI or through the following CLI tools:pct restoreContainer restore utilityqmrestoreVirtual Machine restore utilityFor details see the corresponding manual pages.15.3.1 Bandwidth LimitRestoring one or more big backups may need a lot of resources, especially storage bandwidth for bothreading from the backup storage and writing to the target storage. This can negatively effect other virtualguest as access to storage can get congested.To avoid this you can set bandwidth limits for a backup job. Proxmox VE implements two kinds of limits forrestoring and archive:• per-restore limit: denotes the maximal amount of bandwidth for reading from a backup archive• per-storage write limit: denotes the maximal amount of bandwidth used for writing to a specific storageThe read limit indirectly affects the write limit, as we cannot write more than we read. A smaller per-job limitwill overwrite a bigger per-storage limit. A bigger per-job limit will only overwrite the per-storage limit if youhave ‘Data.Allocate’ permissions on the affected storage.You can use the ‘--bwlimit <integer>` option from the restore CLI commands to set up a restore job specificbandwidth limit. Kibit/s is used as unit for the limit, this means passing `10240’ will limit the read speed ofthe backup to 10 MiB/s, ensuring that the rest of the possible storage bandwidth is available for the alreadyrunning virtual guests, and thus the backup does not impact their operations.NoteYou can use ‘0` for the bwlimit parameter to disable all limits for a specific restore job. This can behelpful if you need to restore a very important virtual guest as fast as possible. (Needs `Data.Allocate’permissions on storage) Proxmox VE Administration Guide 212 / 328Most times your storage’s generally available bandwidth stays the same over time, thus we implemented thepossibility to set a default bandwidth limit per configured storage, this can be done with:# pvesm set STORAGEID --bwlimit KIBs15.4 ConfigurationGlobal configuration is stored in /etc/vzdump.conf. The file uses a simple colon separated key/valueformat. Each line has the following format:OPTION: valueBlank lines in the file are ignored, and lines starting with a # character are treated as comments and are alsoignored. Values from this file are used as default, and can be overwritten on the command line.We currently support the following options:bwlimit: <integer> (0 - N) (default = 0)Limit I/O bandwidth (KBytes per second).compress: <0 | 1 | gzip | lzo> (default = 0)Compress dump file.dumpdir: <string>Store resulting files to specified directory.exclude-path: <string>Exclude certain files/directories (shell globs).ionice: <integer> (0 - 8) (default = 7)Set CFQ ionice priority.lockwait: <integer> (0 - N) (default = 180)Maximal time to wait for the global lock (minutes).mailnotification: <always | failure> (default = always)Specify when to send an emailmailto: <string>Comma-separated list of email addresses that should receive email notifications.maxfiles: <integer> (1 - N) (default = 1)Maximal number of backup files per guest system.mode: <snapshot | stop | suspend> (default = snapshot)Backup mode. Proxmox VE Administration Guide 213 / 328pigz: <integer> (default = 0)Use pigz instead of gzip when N>0. N=1 uses half of cores, N>1 uses N as thread count.remove: <boolean> (default = 1)Remove old backup files if there are more than maxfiles backup files.script: <string>Use specified hook script.stdexcludes: <boolean> (default = 1)Exclude temporary files and logs.stopwait: <integer> (0 - N) (default = 10)Maximal time to wait until a guest system is stopped (minutes).storage: <string>Store resulting file to this storage.tmpdir: <string>Store temporary files to specified directory.Example vzdump.conf Configurationtmpdir: /mnt/fast_local_diskstorage: my_backup_storagemode: snapshotbwlimit: 1000015.5 Hook ScriptsYou can specify a hook script with option --script. This script is called at various phases of the backupprocess, with parameters accordingly set. You can find an example in the documentation directory (vzdump-hook-script.pl).15.6 File ExclusionsNotethis option is only available for container backups.vzdump skips the following files by default (disable with the option --stdexcludes 0)/tmp/?*/var/tmp/?*/var/run/?*pid Proxmox VE Administration Guide 214 / 328You can also manually specify (additional) exclude paths, for example:# vzdump 777 --exclude-path /tmp/ --exclude-path ’/var/foo*’(only excludes tmp directories)Configuration files are also stored inside the backup archive (in ./etc/vzdump/) and will be correctlyrestored.15.7 ExamplesSimply dump guest 777 - no snapshot, just archive the guest private area and configuration files to the defaultdump directory (usually /var/lib/vz/dump/).# vzdump 777Use rsync and suspend/resume to create a snapshot (minimal downtime).# vzdump 777 --mode suspendBackup all guest systems and send notification mails to root and admin.# vzdump --all --mode suspend --mailto root --mailto adminUse snapshot mode (no downtime) and non-default dump directory.# vzdump 777 --dumpdir /mnt/backup --mode snapshotBackup more than one guest (selectively)# vzdump 101 102 103 --mailto rootBackup all guests excluding 101 and 102# vzdump --mode suspend --exclude 101,102Restore a container to a new CT 600# pct restore 600 /mnt/backup/vzdump-lxc-777.tarRestore a QemuServer VM to VM 601# qmrestore /mnt/backup/vzdump-qemu-888.vma 601Clone an existing container 101 to a new container 300 with a 4GB root file system, using pipes# vzdump 101 --stdout | pct restore --rootfs 4 300 - Proxmox VE Administration Guide 215 / 328Chapter 16Important Service Daemons16.1 pvedaemon - Proxmox VE API DaemonThis daemon exposes the whole Proxmox VE API on 127.0.0.1:85. It runs as root and has permissionto do all privileged operations.NoteThe daemon listens to a local address only, so you cannot access it from outside. The pveproxydaemon exposes the API to the outside world.16.2 pveproxy - Proxmox VE API Proxy DaemonThis daemon exposes the whole Proxmox VE API on TCP port 8006 using HTTPS. It runs as user www-data and has very limited permissions. Operation requiring more permissions are forwarded to the localpvedaemon.Requests targeted for other nodes are automatically forwarded to those nodes. This means that you canmanage your whole cluster by connecting to a single Proxmox VE node.16.2.1 Host based Access ControlIt is possible to configure “apache2”-like access control lists. Values are read from file /etc/default/pveproxy. For example:ALLOW_FROM="10.0.0.1-10.0.0.5,192.168.0.0/22"DENY_FROM="all"POLICY="allow"IP addresses can be specified using any syntax understood by Net::IP. The name all is an alias for 0/0.The default policy is allow. Proxmox VE Administration Guide 216 / 328Match POLICY=deny POLICY=allowMatch Allow only allow allowMatch Deny only deny denyNo match deny allowMatch Both Allow & Deny deny allow16.2.2 SSL Cipher SuiteYou can define the cipher list in /etc/default/pveproxy, for exampleCIPHERS="HIGH:MEDIUM:!aNULL:!MD5"Above is the default. See the ciphers(1) man page from the openssl package for a list of all available options.16.2.3 Diffie-Hellman ParametersYou can define the used Diffie-Hellman parameters in /etc/default/pveproxy by setting DHPARAMSto the path of a file containing DH parameters in PEM format, for exampleDHPARAMS="/path/to/dhparams.pem"If this option is not set, the built-in skip2048 parameters will be used.NoteDH parameters are only used if a cipher suite utilizing the DH key exchange algorithm is negotiated.16.2.4 Alternative HTTPS certificateYou can change the certificate used to an external one or to one obtained via ACME.pveproxy uses /etc/pve/local/pveproxy-ssl.pem and /etc/pve/local/pveproxy-ssl.key, if present, and falls back to /etc/pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key. The private key may not use a passphrase.See the Host System Administration chapter of the documentation for details.16.3 pvestatd - Proxmox VE Status DaemonThis daemon queries the status of VMs, storages and containers at regular intervals. The result is sent to allnodes in the cluster.16.4 spiceproxy - SPICE Proxy ServiceSPICE (the Simple Protocol for Independent Computing Environments) is an open remote computing solu-tion, providing client access to remote displays and devices (e.g. keyboard, mouse, audio). The main usecase is to get remote access to virtual machines and container.This daemon listens on TCP port 3128, and implements an HTTP proxy to forward CONNECT request fromthe SPICE client to the correct Proxmox VE VM. It runs as user www-data and has very limited permissions. Proxmox VE Administration Guide 10 / 328You normally select Install Proxmox VE to start the installation. After that you get prompted to select thetarget hard disk(s). The Options button lets you select the target file system, which defaults to ext4. Theinstaller uses LVM if you select ext3, ext4 or xfs as file system, and offers additional option to restrictLVM space (see below)If you have more than one disk, you can also use ZFS as file system. ZFS supports several software RAIDlevels, so this is specially useful if you do not have a hardware RAID controller. The Options button letsyou select the ZFS RAID level, and you can choose disks there. Proxmox VE Administration Guide 217 / 32816.4.1 Host based Access ControlIt is possible to configure "apache2" like access control lists. Values are read from file /etc/default/pveproxy. See pveproxy documentation for details. Proxmox VE Administration Guide 218 / 328Chapter 17Useful Command Line Tools17.1 pvesubscription - Subscription ManagementThis tool is used to handle Proxmox VE subscriptions.17.2 pveperf - Proxmox VE Benchmark ScriptTries to gather some CPU/hard disk performance data on the hard disk mounted at PATH (/ is used asdefault):CPU BOGOMIPSbogomips sum of all CPUsREGEX/SECONDregular expressions per second (perl performance test), should be above 300000HD SIZEhard disk sizeBUFFERED READSsimple HD read test. Modern HDs should reach at least 40 MB/secAVERAGE SEEK TIMEtests average seek time. Fast SCSI HDs reach values < 8 milliseconds. Common IDE/SATA disks getvalues from 15 to 20 ms.FSYNCS/SECONDvalue should be greater than 200 (you should enable write back cache mode on you RAID con-troller - needs a battery backed cache (BBWC)).DNS EXTaverage time to resolve an external DNS name Proxmox VE Administration Guide 219 / 328DNS INTaverage time to resolve a local DNS name Proxmox VE Administration Guide 220 / 328Chapter 18Frequently Asked QuestionsNoteNew FAQs are appended to the bottom of this section.1. What distribution is Proxmox VE based on?Proxmox VE is based on Debian GNU/Linux2. What license does the Proxmox VE project use?Proxmox VE code is licensed under the GNU Affero General Public License, version 3.3. Will Proxmox VE run on a 32bit processor?Proxmox VE works only on 64-bit CPUs (AMD or Intel). There is no plan for 32-bit for the platform.NoteVMs and Containers can be both 32-bit and/or 64-bit.4. Does my CPU support virtualization?To check if your CPU is virtualization compatible, check for the vmx or svm tag in this commandoutput:egrep ’(vmx|svm)’ /proc/cpuinfo5. Supported Intel CPUs64-bit processors with Intel Virtualization Technology (Intel VT-x) support. (List of processors with IntelVT and 64-bit)6. Supported AMD CPUs64-bit processors with AMD Virtualization Technology (AMD-V) support.7. What is a container, CT, VE, Virtual Private Server, VPS?Operating-system-level virtualization is a server-virtualization method where the kernel of an operatingsystem allows for multiple isolated user-space instances, instead of just one. We call such instancescontainers. As containers use the host’s kernel they are limited to Linux guests. Proxmox VE Administration Guide 221 / 3288. What is a QEMU/KVM guest (or VM)?A QEMU/KVM guest (or VM) is a guest system running virtualized under Proxmox VE using QEMUand the Linux KVM kernel module.9. What is QEMU?QEMU is a generic and open source machine emulator and virtualizer. QEMU uses the Linux KVMkernel module to achieve near native performance by executing the guest code directly on the hostCPU. It is not limited to Linux guests but allows arbitrary operating systems to run.10. How long will my Proxmox VE version be supported?Proxmox VE versions are supported at least as long as the corresponding Debian Version is oldstable.Proxmox VE uses a rolling release model and using the latest stable version is always recommended.Proxmox VEVersionDebian Version First Release Debian EOL Proxmox EOLProxmox VE 5.x Debian 9(Stretch)2017-07 tba tbaProxmox VE 4.x Debian 8(Jessie)2015-10 2018-06 2018-06Proxmox VE 3.x Debian 7(Wheezy)2013-05 2016-04 2017-02Proxmox VE 2.x Debian 6(Squeeze)2012-04 2014-05 2014-05Proxmox VE 1.x Debian 5(Lenny)2008-10 2012-03 2013-0111. LXC vs LXD vs Proxmox Containers vs DockerLXC is a userspace interface for the Linux kernel containment features. Through a powerful APIand simple tools, it lets Linux users easily create and manage system containers. LXC, as well asthe former OpenVZ, aims at system virtualization, i.e. allows you to run a complete OS inside acontainer, where you log in as ssh, add users, run apache, etc. . .LXD is building on top of LXC to provide a new, better user experience. Under the hood, LXD usesLXC through liblxc and its Go binding to create and manage the containers. It’s basically analternative to LXC’s tools and distribution template system with the added features that come frombeing controllable over the network.Proxmox Containers also aims at system virtualization, and thus uses LXC as the basis of its owncontainer offer. The Proxmox Container Toolkit is called pct, and is tightly coupled with Proxmox VE.That means that it is aware of the cluster setup, and it can use the same network and storage resourcesas fully virtualized VMs. You can even use the Proxmox VE firewall, create and restore backups, ormanage containers using the HA framework. Everything can be controlled over the network using theProxmox VE API.Docker aims at running a single application running in a contained environment. Hence you’re man-aging a docker instance from the host with the docker toolkit. It is not recommended to run dockerdirectly on your Proxmox VE host.NoteYou can however perfectly install and use docker inside a Proxmox Qemu VM, and thus getting thebenefit of software containerization with the very strong isolation that VMs provide. Proxmox VE Administration Guide 222 / 328 Proxmox VE Administration Guide 223 / 328Chapter 19Bibliography19.1 Books about Proxmox VE[1] [Ahmed16] Wasim Ahmed. Mastering Proxmox - Third Edition. Packt Publishing, 2017. ISBN978-1788397605[2] [Ahmed15] Wasim Ahmed. Proxmox Cookbook. Packt Publishing, 2015. ISBN 978-1783980901[3] [Cheng14] Simon M.C. Cheng. Proxmox High Availability. Packt Publishing, 2014. ISBN 978-1783980888[4] [Goldman16] Rik Goldman. Learning Proxmox VE. Packt Publishing, 2016. ISBN 978-1783981786[5] [Surber16]] Lee R. Surber. Virtualization Complete: Business Basic Edition. Linux Solutions(LRS-TEK), 2016. ASIN B01BBVQZT619.2 Books about related technology[6] [Hertzog13] Raphaël Hertzog & Roland Mas. The Debian Administrator’s Handbook: DebianJessie from Discovery to Mastery, Freexian, 2013. ISBN 979-1091414050[7] [Bir96] Kenneth P. Birman. Building Secure and Reliable Network Applications. Manning Pub-lications Co, 1996. ISBN 978-1884777295[8] [Walsh10] Norman Walsh. DocBook 5: The Definitive Guide. O’Reilly & Associates, 2010.ISBN 978-0596805029[9] [Richardson07] Leonard Richardson & Sam Ruby. RESTful Web Services. O’Reilly Media,2007. ISBN 978-0596529260[10] [Singh15] Karan Singh. Learning Ceph. Packt Publishing, 2015. ISBN 978-1783985623[11] [Singh16] Karan Signh. Ceph Cookbook Packt Publishing, 2016. ISBN 978-1784393502 Proxmox VE Administration Guide 224 / 328[12] [Mauerer08] Wolfgang Mauerer. Professional Linux Kernel Architecture. John Wiley & Sons,2008. ISBN 978-0470343432[13] [Loshin03] Pete Loshin, IPv6: Theory, Protocol, and Practice, 2nd Edition. Morgan Kauf-mann, 2003. ISBN 978-1558608108[14] [Loeliger12] Jon Loeliger & Matthew McCullough. Version Control with Git: Powerful toolsand techniques for collaborative software development. O’Reilly and Associates, 2012. ISBN978-1449316389[15] [Kreibich10] Jay A. Kreibich. Using SQLite, O’Reilly and Associates, 2010. ISBN 978-059652118919.3 Books about related topics[16] [Bessen09] James Bessen & Michael J. Meurer, Patent Failure: How Judges, Bureaucrats,and Lawyers Put Innovators at Risk. Princeton Univ Press, 2009. ISBN 978-0691143217 Proxmox VE Administration Guide 225 / 328Appendix ACommand Line InterfaceA.1 pvesm - Proxmox VE Storage Managerpvesm <COMMAND> [ARGS] [OPTIONS]pvesm add <type> <storage> [OPTIONS]Create a new storage.<type>: <cifs | dir | drbd | glusterfs | iscsi | iscsidirect | lvm |lvmthin | nfs | rbd | sheepdog | zfs | zfspool>Storage type.<storage>: <string>The storage identifier.--authsupported <string>Authsupported.--base <string>Base volume. This volume is automatically activated.--blocksize <string>block size--bwlimit [clone=<LIMIT>] [,default=<LIMIT>] [,migration=<LIMIT>][,move=<LIMIT>] [,restore=<LIMIT>]Set bandwidth/io limits various operations.--comstar_hg <string>host group for comstar views--comstar_tg <string>target group for comstar views Proxmox VE Administration Guide 226 / 328--content <string>Allowed content types.Notethe value rootdir is used for Containers, and value images for VMs.--disable <boolean>Flag to disable the storage.--domain <string>CIFS domain.--export <string>NFS export path.--format <string>Default image format.--is_mountpoint <string> (default = no)Assume the given path is an externally managed mountpoint and consider the storage offline if it isnot mounted. Using a boolean (yes/no) value serves as a shortcut to using the target path in this field.--iscsiprovider <string>iscsi provider--krbd <boolean>Access rbd through krbd kernel module.--maxfiles <integer> (0 - N)Maximal number of backup files per VM. Use 0 for unlimted.--mkdir <boolean> (default = yes)Create the directory if it doesn’t exist.--monhost <string>IP addresses of monitors (for external clusters).--nodes <string>List of cluster node names.--nowritecache <boolean>disable write caching on the target--options <string>NFS mount options (see man nfs) Proxmox VE Administration Guide 11 / 328The next page just ask for basic configuration options like your location, the time zone and keyboard layout.The location is used to select a download server near you to speedup updates. The installer is usually ableto auto detect those setting, so you only need to change them in rare situations when auto detection fails, orwhen you want to use some special keyboard layout not commonly used in your country. Proxmox VE Administration Guide 227 / 328--passwordPassword for CIFS share.--path <string>File system path.--pool <string>Pool.--portal <string>iSCSI portal (IP or DNS name with optional port).--redundancy <integer> (1 - 16) (default = 2)The redundancy count specifies the number of nodes to which the resource should be deployed. Itmust be at least 1 and at most the number of nodes in the cluster.--saferemove <boolean>Zero-out data when removing LVs.--saferemove_throughput <string>Wipe throughput (cstream -t parameter value).--server <string>Server IP or DNS name.--server2 <string>Backup volfile server IP or DNS name.NoteRequires option(s): server--share <string>CIFS share.--shared <boolean>Mark storage as shared.--smbversion <string>no description available--sparse <boolean>use sparse volumes--tagged_only <boolean>Only use logical volumes tagged with pve-vm-ID. Proxmox VE Administration Guide 228 / 328--target <string>iSCSI target.--thinpool <string>LVM thin pool LV name.--transport <rdma | tcp | unix>Gluster transport: tcp or rdma--username <string>RBD Id.--vgname <string>Volume group name.--volume <string>Glusterfs Volume.pvesm alloc <storage> <vmid> <filename> <size> [OPTIONS]Allocate disk images.<storage>: <string>The storage identifier.<vmid>: <integer> (1 - N)Specify owner VM<filename>: <string>The name of the file to create.<size>: \d+[MG]?Size in kilobyte (1024 bytes). Optional suffixes M (megabyte, 1024K) and G (gigabyte, 1024M)--format <qcow2 | raw | subvol>no description availableNoteRequires option(s): sizepvesm cifsscan <server> [OPTIONS]Scan remote CIFS server.<server>: <string>no description available Proxmox VE Administration Guide 229 / 328--domain <string>no description available--passwordno description available--username <string>no description availablepvesm export <volume> <format> <filename> [OPTIONS]Export a volume.<volume>: <string>Volume identifier<format>: <qcow2+size | raw+size | tar+size | vmdk+size | zfs>Export stream format<filename>: <string>Destination file name--base (?ˆ:[a-z0-9_\-]{1,40})Snapshot to start an incremental stream from--snapshot (?ˆ:[a-z0-9_\-]{1,40})Snapshot to export--with-snapshots <boolean> (default = 0)Whether to include intermediate snapshots in the streampvesm extractconfig <volume>Extract configuration from vzdump backup archive.<volume>: <string>Volume identifierpvesm free <volume> [OPTIONS]Delete volume<volume>: <string>Volume identifier--storage <string>The storage identifier. Proxmox VE Administration Guide 230 / 328pvesm glusterfsscan <server>Scan remote GlusterFS server.<server>: <string>no description availablepvesm help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pvesm import <volume> <format> <filename> [OPTIONS]Import a volume.<volume>: <string>Volume identifier<format>: <qcow2+size | raw+size | tar+size | vmdk+size | zfs>Import stream format<filename>: <string>Source file name--base (?ˆ:[a-z0-9_\-]{1,40})Base snapshot of an incremental stream--delete-snapshot (?ˆ:[a-z0-9_\-]{1,80})A snapshot to delete on success--with-snapshots <boolean> (default = 0)Whether the stream includes intermediate snapshotspvesm iscsiscan --portal <string> [OPTIONS]Scan remote iSCSI server.--portal <string>no description availablepvesm list <storage> [OPTIONS]List storage content. Proxmox VE Administration Guide 231 / 328<storage>: <string>The storage identifier.--content <string>Only list content of this type.--vmid <integer> (1 - N)Only list images for this VMpvesm lvmscanList local LVM volume groups.pvesm lvmthinscan <vg>List local LVM Thin Pools.<vg>: [a-zA-Z0-9\.\+\_][a-zA-Z0-9\.\+\_\-]+no description availablepvesm nfsscan <server>Scan remote NFS server.<server>: <string>no description availablepvesm path <volume>Get filesystem path for specified volume<volume>: <string>Volume identifierpvesm remove <storage>Delete storage configuration.<storage>: <string>The storage identifier.pvesm set <storage> [OPTIONS]Update storage configuration.<storage>: <string>The storage identifier.--blocksize <string>block size Proxmox VE Administration Guide 232 / 328--bwlimit [clone=<LIMIT>] [,default=<LIMIT>] [,migration=<LIMIT>][,move=<LIMIT>] [,restore=<LIMIT>]Set bandwidth/io limits various operations.--comstar_hg <string>host group for comstar views--comstar_tg <string>target group for comstar views--content <string>Allowed content types.Notethe value rootdir is used for Containers, and value images for VMs.--delete <string>A list of settings you want to delete.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--disable <boolean>Flag to disable the storage.--domain <string>CIFS domain.--format <string>Default image format.--is_mountpoint <string> (default = no)Assume the given path is an externally managed mountpoint and consider the storage offline if it isnot mounted. Using a boolean (yes/no) value serves as a shortcut to using the target path in this field.--krbd <boolean>Access rbd through krbd kernel module.--maxfiles <integer> (0 - N)Maximal number of backup files per VM. Use 0 for unlimted.--mkdir <boolean> (default = yes)Create the directory if it doesn’t exist. Proxmox VE Administration Guide 233 / 328--monhost <string>IP addresses of monitors (for external clusters).--nodes <string>List of cluster node names.--nowritecache <boolean>disable write caching on the target--options <string>NFS mount options (see man nfs)--passwordPassword for CIFS share.--pool <string>Pool.--redundancy <integer> (1 - 16) (default = 2)The redundancy count specifies the number of nodes to which the resource should be deployed. Itmust be at least 1 and at most the number of nodes in the cluster.--saferemove <boolean>Zero-out data when removing LVs.--saferemove_throughput <string>Wipe throughput (cstream -t parameter value).--server <string>Server IP or DNS name.--server2 <string>Backup volfile server IP or DNS name.NoteRequires option(s): server--shared <boolean>Mark storage as shared.--smbversion <string>no description available--sparse <boolean>use sparse volumes Proxmox VE Administration Guide 234 / 328--tagged_only <boolean>Only use logical volumes tagged with pve-vm-ID.--transport <rdma | tcp | unix>Gluster transport: tcp or rdma--username <string>RBD Id.pvesm status [OPTIONS]Get status for all datastores.--content <string>Only list stores which support this content type.--enabled <boolean> (default = 0)Only list stores which are enabled (not disabled in config).--format <boolean> (default = 0)Include information about formats--storage <string>Only list status for specified storage--target <string>If target is different to node, we only lists shared storages which content is accessible on this node andthe specified target node.pvesm zfsscanScan zfs pool list on local node.A.2 pvesubscription - Proxmox VE Subscription Managerpvesubscription <COMMAND> [ARGS] [OPTIONS]pvesubscription getRead subscription info.pvesubscription help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format. Proxmox VE Administration Guide 235 / 328pvesubscription set <key>Set subscription key.<key>: pve([124])([cbsp])-[0-9a-f]{10}Proxmox VE subscription keypvesubscription update [OPTIONS]Update subscription info.--force <boolean> (default = 0)Always connect to server, even if we have up to date info inside local cache.A.3 pveperf - Proxmox VE Benchmark Scriptpveperf [PATH]A.4 pveceph - Manage CEPH Services on Proxmox VE Nodespveceph <COMMAND> [ARGS] [OPTIONS]pveceph createmgr [OPTIONS]Create Ceph Manager--id [a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?The ID for the manager, when omitted the same as the nodenamepveceph createmon [OPTIONS]Create Ceph Monitor and Manager--exclude-manager <boolean> (default = 0)When set, only a monitor will be created.--id [a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?The ID for the monitor, when omitted the same as the nodename--mon-address <string>Overwrites autodetected monitor IP address. Must be in the public network of ceph.pveceph createosd <dev> [OPTIONS]Create OSD<dev>: <string>Block device name. Proxmox VE Administration Guide 236 / 328--bluestore <boolean> (default = 1)Use bluestore instead of filestore. This is the default.--fstype <btrfs | ext4 | xfs> (default = xfs)File system type (filestore only).--journal_dev <string>Block device name for journal (filestore) or block.db (bluestore).--wal_dev <string>Block device name for block.wal (bluestore only).pveceph createpool <name> [OPTIONS]Create POOL<name>: <string>The name of the pool. It must be unique.--add_storages <boolean>Configure VM and CT storages using the new pool.--application <cephfs | rbd | rgw>The application of the pool, rbd by default.--crush_rule <string>The rule to use for mapping object placement in the cluster.--min_size <integer> (1 - 7) (default = 2)Minimum number of replicas per object--pg_num <integer> (8 - 32768) (default = 64)Number of placement groups.--size <integer> (1 - 7) (default = 3)Number of replicas per objectpveceph destroymgr <id>Destroy Ceph Manager.<id>: [a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?The ID of the managerpveceph destroymon <monid> [OPTIONS]Destroy Ceph Monitor and Manager. Proxmox VE Administration Guide 12 / 328You then need to specify an email address and the superuser (root) password. The password must have atleast 5 characters, but we highly recommend to use stronger passwords - here are some guidelines:• Use a minimum password length of 12 to 14 characters.• Include lowercase and uppercase alphabetic characters, numbers and symbols.• Avoid character repetition, keyboard patterns, dictionary words, letter or number sequences, usernames,relative or pet names, romantic links (current or past) and biographical information (e.g., ID numbers,ancestors’ names or dates).It is sometimes necessary to send notification to the system administrator, for example:• Information about available package updates.• Error messages from periodic CRON jobs.All those notification mails will be sent to the specified email address. Proxmox VE Administration Guide 237 / 328<monid>: [a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?Monitor ID--exclude-manager <boolean> (default = 0)When set, removes only the monitor, not the managerpveceph destroyosd <osdid> [OPTIONS]Destroy OSD<osdid>: <integer>OSD ID--cleanup <boolean> (default = 0)If set, we remove partition table entries.pveceph destroypool <name> [OPTIONS]Destroy pool<name>: <string>The name of the pool. It must be unique.--force <boolean> (default = 0)If true, destroys pool even if in use--remove_storages <boolean> (default = 0)Remove all pveceph-managed storages configured for this poolpveceph help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pveceph init [OPTIONS]Create initial ceph default configuration and setup symlinks.--disable_cephx <boolean> (default = 0)Disable cephx authentification.Warningcephx is a security feature protecting against man-in-the-middle attacks. Only consider dis-abling cephx if your network is private! Proxmox VE Administration Guide 238 / 328--min_size <integer> (1 - 7) (default = 2)Minimum number of available replicas per object to allow I/O--network <string>Use specific network for all ceph related traffic--pg_bits <integer> (6 - 14) (default = 6)Placement group bits, used to specify the default number of placement groups.Noteosd pool default pg num does not work for default pools.--size <integer> (1 - 7) (default = 3)Targeted number of replicas per objectpveceph install [OPTIONS]Install ceph related packages.--version <luminous>no description availablepveceph lspoolsList all pools.pveceph purgeDestroy ceph related data and configuration files.pveceph start [<service>]Start ceph services.<service>: (mon|mds|osd|mgr)\.[A-Za-z0-9\-]{1,32}Ceph service name.pveceph statusGet ceph status.pveceph stop [<service>]Stop ceph services.<service>: (mon|mds|osd|mgr)\.[A-Za-z0-9\-]{1,32}Ceph service name. Proxmox VE Administration Guide 239 / 328A.5 pvenode - Proxmox VE Node Managementpvenode <COMMAND> [ARGS] [OPTIONS]pvenode acme account deactivate [<name>]Deactivate existing ACME account at CA.<name>: <name> (default = default)ACME account config file name.pvenode acme account info [<name>]Return existing ACME account information.<name>: <name> (default = default)ACME account config file name.pvenode acme account listACMEAccount index.pvenode acme account register [<name>] {<contact>} [OPTIONS]Register a new ACME account with a compatible CA.<name>: <name> (default = default)ACME account config file name.<contact>: <string>Contact email addresses.--directory ˆhttps?://.*URL of ACME CA directory endpoint.pvenode acme account update [<name>] [OPTIONS]Update existing ACME account information with CA. Note: not specifying any new account informationtriggers a refresh.<name>: <name> (default = default)ACME account config file name.--contact <string>Contact email addresses.pvenode acme cert order [OPTIONS]Order a new certificate from ACME-compatible CA. Proxmox VE Administration Guide 240 / 328--force <boolean> (default = 0)Overwrite existing custom certificate.pvenode acme cert renew [OPTIONS]Renew existing certificate from CA.--force <boolean> (default = 0)Force renewal even if expiry is more than 30 days away.pvenode acme cert revokeRevoke existing certificate from CA.pvenode cert delete [<restart>]DELETE custom certificate chain and key.<restart>: <boolean> (default = 0)Restart pveproxy.pvenode cert infoGet information about node’s certificates.pvenode cert set <certificates> [<key>] [OPTIONS]Upload or update custom certificate chain and key.<certificates>: <string>PEM encoded certificate (chain).<key>: <string>PEM encoded private key.--force <boolean> (default = 0)Overwrite existing custom or ACME certificate files.--restart <boolean> (default = 0)Restart pveproxy.pvenode config getGet node configuration options.pvenode config set [OPTIONS]Set node configuration options.--acme domains=<domain[;domain;...]> [,account=<name>]Node specific ACME settings. Proxmox VE Administration Guide 241 / 328--delete <string>A list of settings you want to delete.--description <string>Node description/comment.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.pvenode help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.A.6 qm - Qemu/KVM Virtual Machine Managerqm <COMMAND> [ARGS] [OPTIONS]qm agent <vmid> <command>Execute Qemu Guest Agent commands.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<command>: <fsfreeze-freeze | fsfreeze-status | fsfreeze-thaw |fstrim | get-fsinfo | get-host-name | get-memory-block-info | get-memory-blocks | get-osinfo | get-time | get-timezone | get-users |get-vcpus | info | network-get-interfaces | ping | shutdown |suspend-disk | suspend-hybrid | suspend-ram>The QGA command.qm clone <vmid> <newid> [OPTIONS]Create a copy of virtual machine/template.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<newid>: <integer> (1 - N)VMID for the clone. Proxmox VE Administration Guide 242 / 328--description <string>Description for the new VM.--format <qcow2 | raw | vmdk>Target format for file storage. Only valid for full clone.--full <boolean>Create a full copy of all disks. This is always done when you clone a normal VM. For VM templates,we try to create a linked clone by default.--name <string>Set a name for the new VM.--pool <string>Add the new VM to the specified pool.--snapname <string>The name of the snapshot.--storage <string>Target storage for full clone.--target <string>Target node. Only allowed if the original VM is on shared storage.qm config <vmid> [OPTIONS]Get current virtual machine configuration. This does not include pending configuration changes (see pendingAPI).<vmid>: <integer> (1 - N)The (unique) ID of the VM.--current <boolean> (default = 0)Get current values (instead of pending values).qm create <vmid> [OPTIONS]Create or restore a virtual machine.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--acpi <boolean> (default = 1)Enable/disable ACPI.--agent <boolean> (default = 0)Enable/disable Qemu GuestAgent. Proxmox VE Administration Guide 243 / 328--archive <string>The backup file.--args <string>Arbitrary arguments passed to kvm.--autostart <boolean> (default = 0)Automatic restart after crash (currently ignored).--balloon <integer> (0 - N)Amount of target RAM for the VM in MB. Using zero disables the ballon driver.--bios <ovmf | seabios> (default = seabios)Select BIOS implementation.--boot [acdn]{1,4} (default = cdn)Boot on floppy (a), hard disk (c), CD-ROM (d), or network (n).--bootdisk (ide|sata|scsi|virtio)\d+Enable booting from specified disk.--bwlimit <integer> (0 - N)Override i/o bandwidth limit (in KiB/s).--cdrom <volume>This is an alias for option -ide2--cipassword <password>cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keysinstead. Also note that older cloud-init versions do not support hashed passwords.--citype <configdrive2 | nocloud>Specifies the cloud-init configuration format. The default depends on the configured operating systemtype (ostype. We use the nocloud format for Linux, and configdrive2 for windows.--ciuser <string>cloud-init: User name to change ssh keys and password for instead of the image’s configured defaultuser.--cores <integer> (1 - N) (default = 1)The number of cores per socket.--cpu [cputype=]<enum> [,flags=<+FLAG[;-FLAG...]>] [,hidden=<1|0>]Emulated CPU type. Proxmox VE Administration Guide 244 / 328--cpulimit <number> (0 - 128) (default = 0)Limit of CPU usage.--cpuunits <integer> (2 - 262144) (default = 1024)CPU weight for a VM.--description <string>Description for the VM. Only used on the configuration web interface. This is saved as comment insidethe configuration file.--efidisk0 [file=]<volume> [,format=<enum>] [,size=<DiskSize>]Configure a Disk for storing EFI vars--force <boolean>Allow to overwrite existing VM.NoteRequires option(s): archive--freeze <boolean>Freeze CPU at startup (use c monitor command to start execution).--hostpci[n] [host=]<HOSTPCIID[;HOSTPCIID2...]> [,pcie=<1|0>][,rombar=<1|0>] [,romfile=<string>] [,x-vga=<1|0>]Map host PCI devices into guest.--hotplug <string> (default = network,disk,usb)Selectively enable hotplug features. This is a comma separated list of hotplug features: network,disk, cpu, memory and usb. Use 0 to disable hotplug completely. Value 1 is an alias for the defaultnetwork,disk,usb.--hugepages <1024 | 2 | any>Enable/disable hugepages memory. Proxmox VE Administration Guide 245 / 328--ide[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,mbps=<mbps>] [,mbps_max=<mbps>] [,mbps_rd=<mbps>][,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>] [,mbps_wr_max=<mbps>][,media=<cdrom|disk>] [,model=<model>] [,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as IDE hard disk or CD-ROM (n is 0 to 3).--ipconfig[n] [gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,ip=<IPv4Format/CIDR>] [,ip6=<IPv6Format/CIDR>]cloud-init: Specify IP addresses and gateways for the corresponding interface.IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.The special string dhcp can be used for IP addresses to use DHCP, in which case no explicit gatewayshould be provided. For IPv6 the special string auto can be used to use stateless autoconfiguration.If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using dhcpon IPv4.--keyboard <da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be| fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt |pt-br | sl | sv | tr>Keybord layout for vnc server. Default is read from the /etc/pve/datacenter.conf configuration file.Itshould not be necessary to set it.--kvm <boolean> (default = 1)Enable/disable KVM hardware virtualization.--localtime <boolean>Set the real time clock to local time. This is enabled by default if ostype indicates a Microsoft OS.--lock <backup | migrate | rollback | snapshot>Lock/unlock the VM.--machine (pc|pc(-i440fx)?-\d+\.\d+(\.pxe)?|q35|pc-q35-\d+\.\d+(\.pxe)?)Specific the Qemu machine type.--memory <integer> (16 - N) (default = 512)Amount of RAM for the VM in MB. This is the maximum available memory when you use the balloondevice. Proxmox VE Administration Guide 246 / 328--migrate_downtime <number> (0 - N) (default = 0.1)Set maximum tolerated downtime (in seconds) for migrations.--migrate_speed <integer> (0 - N) (default = 0)Set maximum speed (in MB/s) for migrations. Value 0 is no limit.--name <string>Set a name for the VM. Only used on the configuration web interface.--nameserver <string>cloud-init: Sets DNS server IP address for a container. Create will automatically use the setting fromthe host if neither searchdomain nor nameserver are set.--net[n] [model=]<enum> [,bridge=<bridge>] [,firewall=<1|0>][,link_down=<1|0>] [,macaddr=<XX:XX:XX:XX:XX:XX>] [,queues=<integer>] [,rate=<number>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,<model>=<macaddr>]Specify network devices.--numa <boolean> (default = 0)Enable/disable NUMA.--numa[n] cpus=<id[-id];...> [,hostnodes=<id[-id];...>] [,memory=<number>] [,policy=<preferred|bind|interleave>]NUMA topology.--onboot <boolean> (default = 0)Specifies whether a VM will be started during system bootup.--ostype <l24 | l26 | other | solaris | w2k | w2k3 | w2k8 | win10 |win7 | win8 | wvista | wxp>Specify guest operating system.--parallel[n] /dev/parport\d+|/dev/usb/lp\d+Map host parallel devices (n is 0 to 2).--pool <string>Add the VM to the specified pool.--protection <boolean> (default = 0)Sets the protection flag of the VM. This will disable the remove VM and remove disk operations.--reboot <boolean> (default = 1)Allow reboot. If set to 0 the VM exit on reboot. Proxmox VE Administration Guide 13 / 328The last step is the network configuration. Please note that you can use either IPv4 or IPv6 here, but notboth. If you want to configure a dual stack node, you can easily do that after installation. Proxmox VE Administration Guide 247 / 328--sata[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,mbps=<mbps>] [,mbps_max=<mbps>] [,mbps_rd=<mbps>][,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>] [,mbps_wr_max=<mbps>][,media=<cdrom|disk>] [,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as SATA hard disk or CD-ROM (n is 0 to 5).--scsi[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,iothread=<1|0>] [,mbps=<mbps>] [,mbps_max=<mbps>][,mbps_rd=<mbps>] [,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>][,mbps_wr_max=<mbps>] [,media=<cdrom|disk>] [,queues=<integer>][,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,scsiblock=<1|0>] [,secs=<integer>] [,serial=<serial>] [,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as SCSI hard disk or CD-ROM (n is 0 to 13).--scsihw <lsi | lsi53c810 | megasas | pvscsi | virtio-scsi-pci |virtio-scsi-single> (default = lsi)SCSI controller model--searchdomain <string>cloud-init: Sets DNS search domains for a container. Create will automatically use the setting from thehost if neither searchdomain nor nameserver are set.--serial[n] (/dev/.+|socket)Create a serial device inside the VM (n is 0 to 3)--shares <integer> (0 - 50000) (default = 1000)Amount of memory shares for auto-ballooning. The larger the number is, the more memory this VMgets. Number is relative to weights of all other running VMs. Using zero disables auto-ballooning Proxmox VE Administration Guide 248 / 328--smbios1 [family=<string>] [,manufacturer=<string>] [,product=<string>] [,serial=<string>] [,sku=<string>] [,uuid=<UUID>][,version=<string>]Specify SMBIOS type 1 fields.--smp <integer> (1 - N) (default = 1)The number of CPUs. Please use option -sockets instead.--sockets <integer> (1 - N) (default = 1)The number of CPU sockets.--sshkeys <filepath>cloud-init: Setup public SSH keys (one key per line, OpenSSH format).--startdate (now | YYYY-MM-DD | YYYY-MM-DDTHH:MM:SS) (default = now)Set the initial date of the real time clock. Valid format for date are: now or 2006-06-17T16:01:21 or2006-06-17.--startup `[[order=]\d+] [,up=\d+] [,down=\d+] `Startup and shutdown behavior. Order is a non-negative number defining the general startup order.Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds,which specifies a delay to wait before the next VM is started or stopped.--storage <string>Default storage.--tablet <boolean> (default = 1)Enable/disable the USB tablet device.--tdf <boolean> (default = 0)Enable/disable time drift fix.--template <boolean> (default = 0)Enable/disable Template.--unique <boolean>Assign a unique random ethernet address.NoteRequires option(s): archive--unused[n] <string>Reference to unused volumes. This is used internally, and should not be modified manually. Proxmox VE Administration Guide 249 / 328--usb[n] [host=]<HOSTUSBDEVICE|spice> [,usb3=<1|0>]Configure an USB device (n is 0 to 4).--vcpus <integer> (1 - N) (default = 0)Number of hotplugged vcpus.--vga <cirrus | qxl | qxl2 | qxl3 | qxl4 | serial0 | serial1 |serial2 | serial3 | std | vmware>Select the VGA type.--virtio[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,iothread=<1|0>] [,mbps=<mbps>] [,mbps_max=<mbps>][,mbps_rd=<mbps>] [,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>][,mbps_wr_max=<mbps>] [,media=<cdrom|disk>] [,replicate=<1|0>][,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as VIRTIO hard disk (n is 0 to 15).--vmstatestorage <string>Default storage for VM state volumes/files.--watchdog [[model=]<i6300esb|ib700>] [,action=<enum>]Create a virtual hardware watchdog device.qm delsnapshot <vmid> <snapname> [OPTIONS]Delete a VM snapshot.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<snapname>: <string>The name of the snapshot.--force <boolean>For removal from config file, even if removing disk snapshots fails.qm destroy <vmid> [OPTIONS]Destroy the vm (also delete all used/owned volumes). Proxmox VE Administration Guide 250 / 328<vmid>: <integer> (1 - N)The (unique) ID of the VM.--skiplock <boolean>Ignore locks - only root is allowed to use this option.qm help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.qm importdisk <vmid> <source> <storage> [OPTIONS]Import an external disk image as an unused disk in a VM. The image format has to be supported by qemu-img(1).<vmid>: <integer> (1 - N)The (unique) ID of the VM.<source>: <string>Path to the disk image to import<storage>: <string>Target storage ID--format <qcow2 | raw | vmdk>Target formatqm importovf <vmid> <manifest> <storage> [OPTIONS]Create a new VM using parameters read from an OVF manifest<vmid>: <integer> (1 - N)The (unique) ID of the VM.<manifest>: <string>path to the ovf file<storage>: <string>Target storage ID--dryrun <boolean>Print a parsed representation of the extracted OVF parameters, but do not create a VM Proxmox VE Administration Guide 251 / 328--format <qcow2 | raw | vmdk>Target formatqm list [OPTIONS]Virtual machine index (per node).--full <boolean>Determine the full status of active VMs.qm listsnapshot <vmid>List all snapshots.<vmid>: <integer> (1 - N)The (unique) ID of the VM.qm migrate <vmid> <target> [OPTIONS]Migrate virtual machine. Creates a new migration task.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<target>: <string>Target node.--force <boolean>Allow to migrate VMs which use local devices. Only root may use this option.--migration_network <string>CIDR of the (sub) network that is used for migration.--migration_type <insecure | secure>Migration traffic is encrypted using an SSH tunnel by default. On secure, completely private networksthis can be disabled to increase performance.--online <boolean>Use online/live migration.--targetstorage <string>Default target storage.--with-local-disks <boolean>Enable live storage migration for local diskqm monitor <vmid>Enter Qemu Monitor interface. Proxmox VE Administration Guide 252 / 328<vmid>: <integer> (1 - N)The (unique) ID of the VM.qm move_disk <vmid> <disk> <storage> [OPTIONS]Move volume to different storage.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<disk>: <efidisk0 | ide0 | ide1 | ide2 | ide3 | sata0 | sata1 |sata2 | sata3 | sata4 | sata5 | scsi0 | scsi1 | scsi10 | scsi11 |scsi12 | scsi13 | scsi2 | scsi3 | scsi4 | scsi5 | scsi6 | scsi7 |scsi8 | scsi9 | virtio0 | virtio1 | virtio10 | virtio11 | virtio12| virtio13 | virtio14 | virtio15 | virtio2 | virtio3 | virtio4 |virtio5 | virtio6 | virtio7 | virtio8 | virtio9>The disk you want to move.<storage>: <string>Target storage.--delete <boolean> (default = 0)Delete the original disk after successful copy. By default the original disk is kept as unused disk.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--format <qcow2 | raw | vmdk>Target Format.qm mtunnelUsed by qmigrate - do not use manually.qm nbdstop <vmid>Stop embedded nbd server.<vmid>: <integer> (1 - N)The (unique) ID of the VM.qm pending <vmid>Get virtual machine configuration, including pending changes.<vmid>: <integer> (1 - N)The (unique) ID of the VM. Proxmox VE Administration Guide 253 / 328qm rescan [OPTIONS]Rescan all storages and update disk sizes and unused disk images.--vmid <integer> (1 - N)The (unique) ID of the VM.qm reset <vmid> [OPTIONS]Reset virtual machine.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--skiplock <boolean>Ignore locks - only root is allowed to use this option.qm resize <vmid> <disk> <size> [OPTIONS]Extend volume size.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<disk>: <efidisk0 | ide0 | ide1 | ide2 | ide3 | sata0 | sata1 |sata2 | sata3 | sata4 | sata5 | scsi0 | scsi1 | scsi10 | scsi11 |scsi12 | scsi13 | scsi2 | scsi3 | scsi4 | scsi5 | scsi6 | scsi7 |scsi8 | scsi9 | virtio0 | virtio1 | virtio10 | virtio11 | virtio12| virtio13 | virtio14 | virtio15 | virtio2 | virtio3 | virtio4 |virtio5 | virtio6 | virtio7 | virtio8 | virtio9>The disk you want to resize.<size>: \+?\d+(\.\d+)?[KMGT]?The new size. With the + sign the value is added to the actual size of the volume and without it, thevalue is taken as an absolute one. Shrinking disk size is not supported.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--skiplock <boolean>Ignore locks - only root is allowed to use this option.qm resume <vmid> [OPTIONS]Resume virtual machine.<vmid>: <integer> (1 - N)The (unique) ID of the VM. Proxmox VE Administration Guide 254 / 328--nocheck <boolean>no description available--skiplock <boolean>Ignore locks - only root is allowed to use this option.qm rollback <vmid> <snapname>Rollback VM state to specified snapshot.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<snapname>: <string>The name of the snapshot.qm sendkey <vmid> <key> [OPTIONS]Send key event to virtual machine.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<key>: <string>The key (qemu monitor encoding).--skiplock <boolean>Ignore locks - only root is allowed to use this option.qm set <vmid> [OPTIONS]Set virtual machine options (synchrounous API) - You should consider using the POST method instead forany actions involving hotplug or storage allocation.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--acpi <boolean> (default = 1)Enable/disable ACPI.--agent <boolean> (default = 0)Enable/disable Qemu GuestAgent.--args <string>Arbitrary arguments passed to kvm.--autostart <boolean> (default = 0)Automatic restart after crash (currently ignored). Proxmox VE Administration Guide 255 / 328--balloon <integer> (0 - N)Amount of target RAM for the VM in MB. Using zero disables the ballon driver.--bios <ovmf | seabios> (default = seabios)Select BIOS implementation.--boot [acdn]{1,4} (default = cdn)Boot on floppy (a), hard disk (c), CD-ROM (d), or network (n).--bootdisk (ide|sata|scsi|virtio)\d+Enable booting from specified disk.--cdrom <volume>This is an alias for option -ide2--cipassword <password>cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keysinstead. Also note that older cloud-init versions do not support hashed passwords.--citype <configdrive2 | nocloud>Specifies the cloud-init configuration format. The default depends on the configured operating systemtype (ostype. We use the nocloud format for Linux, and configdrive2 for windows.--ciuser <string>cloud-init: User name to change ssh keys and password for instead of the image’s configured defaultuser.--cores <integer> (1 - N) (default = 1)The number of cores per socket.--cpu [cputype=]<enum> [,flags=<+FLAG[;-FLAG...]>] [,hidden=<1|0>]Emulated CPU type.--cpulimit <number> (0 - 128) (default = 0)Limit of CPU usage.--cpuunits <integer> (2 - 262144) (default = 1024)CPU weight for a VM.--delete <string>A list of settings you want to delete.--description <string>Description for the VM. Only used on the configuration web interface. This is saved as comment insidethe configuration file. Proxmox VE Administration Guide 256 / 328--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--efidisk0 [file=]<volume> [,format=<enum>] [,size=<DiskSize>]Configure a Disk for storing EFI vars--force <boolean>Force physical removal. Without this, we simple remove the disk from the config file and create anadditional configuration entry called unused[n], which contains the volume ID. Unlink of unused[n]always cause physical removal.NoteRequires option(s): delete--freeze <boolean>Freeze CPU at startup (use c monitor command to start execution).--hostpci[n] [host=]<HOSTPCIID[;HOSTPCIID2...]> [,pcie=<1|0>][,rombar=<1|0>] [,romfile=<string>] [,x-vga=<1|0>]Map host PCI devices into guest.--hotplug <string> (default = network,disk,usb)Selectively enable hotplug features. This is a comma separated list of hotplug features: network,disk, cpu, memory and usb. Use 0 to disable hotplug completely. Value 1 is an alias for the defaultnetwork,disk,usb.--hugepages <1024 | 2 | any>Enable/disable hugepages memory.--ide[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,mbps=<mbps>] [,mbps_max=<mbps>] [,mbps_rd=<mbps>][,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>] [,mbps_wr_max=<mbps>][,media=<cdrom|disk>] [,model=<model>] [,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as IDE hard disk or CD-ROM (n is 0 to 3). Proxmox VE Administration Guide 14 / 328If you press Next now, installation starts to format disks, and copies packages to the target. Please waituntil that is finished, then reboot the server.Further configuration is done via the Proxmox web interface. Just point your browser to the IP address givenduring installation (https://youripaddress:8006).NoteDefault login is "root" (realm PAM) and the root password is defined during the installation process.2.2.1 Advanced LVM Configuration OptionsThe installer creates a Volume Group (VG) called pve, and additional Logical Volumes (LVs) called root,data and swap. The size of those volumes can be controlled with:hdsizeDefines the total HD size to be used. This way you can save free space on the HD for further partition-ing (i.e. for an additional PV and VG on the same hard disk that can be used for LVM storage).swapsizeDefines the size of the swap volume. The default is the size of the installed memory, minimum 4 GBand maximum 8 GB. The resulting value cannot be greater than hdsize/8. Proxmox VE Administration Guide 257 / 328--ipconfig[n] [gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,ip=<IPv4Format/CIDR>] [,ip6=<IPv6Format/CIDR>]cloud-init: Specify IP addresses and gateways for the corresponding interface.IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.The special string dhcp can be used for IP addresses to use DHCP, in which case no explicit gatewayshould be provided. For IPv6 the special string auto can be used to use stateless autoconfiguration.If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using dhcpon IPv4.--keyboard <da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be| fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt |pt-br | sl | sv | tr>Keybord layout for vnc server. Default is read from the /etc/pve/datacenter.conf configuration file.Itshould not be necessary to set it.--kvm <boolean> (default = 1)Enable/disable KVM hardware virtualization.--localtime <boolean>Set the real time clock to local time. This is enabled by default if ostype indicates a Microsoft OS.--lock <backup | migrate | rollback | snapshot>Lock/unlock the VM.--machine (pc|pc(-i440fx)?-\d+\.\d+(\.pxe)?|q35|pc-q35-\d+\.\d+(\.pxe)?)Specific the Qemu machine type.--memory <integer> (16 - N) (default = 512)Amount of RAM for the VM in MB. This is the maximum available memory when you use the balloondevice.--migrate_downtime <number> (0 - N) (default = 0.1)Set maximum tolerated downtime (in seconds) for migrations.--migrate_speed <integer> (0 - N) (default = 0)Set maximum speed (in MB/s) for migrations. Value 0 is no limit.--name <string>Set a name for the VM. Only used on the configuration web interface.--nameserver <string>cloud-init: Sets DNS server IP address for a container. Create will automatically use the setting fromthe host if neither searchdomain nor nameserver are set. Proxmox VE Administration Guide 258 / 328--net[n] [model=]<enum> [,bridge=<bridge>] [,firewall=<1|0>][,link_down=<1|0>] [,macaddr=<XX:XX:XX:XX:XX:XX>] [,queues=<integer>] [,rate=<number>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,<model>=<macaddr>]Specify network devices.--numa <boolean> (default = 0)Enable/disable NUMA.--numa[n] cpus=<id[-id];...> [,hostnodes=<id[-id];...>] [,memory=<number>] [,policy=<preferred|bind|interleave>]NUMA topology.--onboot <boolean> (default = 0)Specifies whether a VM will be started during system bootup.--ostype <l24 | l26 | other | solaris | w2k | w2k3 | w2k8 | win10 |win7 | win8 | wvista | wxp>Specify guest operating system.--parallel[n] /dev/parport\d+|/dev/usb/lp\d+Map host parallel devices (n is 0 to 2).--protection <boolean> (default = 0)Sets the protection flag of the VM. This will disable the remove VM and remove disk operations.--reboot <boolean> (default = 1)Allow reboot. If set to 0 the VM exit on reboot.--revert <string>Revert a pending change.--sata[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,mbps=<mbps>] [,mbps_max=<mbps>] [,mbps_rd=<mbps>][,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>] [,mbps_wr_max=<mbps>][,media=<cdrom|disk>] [,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as SATA hard disk or CD-ROM (n is 0 to 5). Proxmox VE Administration Guide 259 / 328--scsi[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,iothread=<1|0>] [,mbps=<mbps>] [,mbps_max=<mbps>][,mbps_rd=<mbps>] [,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>][,mbps_wr_max=<mbps>] [,media=<cdrom|disk>] [,queues=<integer>][,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,scsiblock=<1|0>] [,secs=<integer>] [,serial=<serial>] [,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as SCSI hard disk or CD-ROM (n is 0 to 13).--scsihw <lsi | lsi53c810 | megasas | pvscsi | virtio-scsi-pci |virtio-scsi-single> (default = lsi)SCSI controller model--searchdomain <string>cloud-init: Sets DNS search domains for a container. Create will automatically use the setting from thehost if neither searchdomain nor nameserver are set.--serial[n] (/dev/.+|socket)Create a serial device inside the VM (n is 0 to 3)--shares <integer> (0 - 50000) (default = 1000)Amount of memory shares for auto-ballooning. The larger the number is, the more memory this VMgets. Number is relative to weights of all other running VMs. Using zero disables auto-ballooning--skiplock <boolean>Ignore locks - only root is allowed to use this option.--smbios1 [family=<string>] [,manufacturer=<string>] [,product=<string>] [,serial=<string>] [,sku=<string>] [,uuid=<UUID>][,version=<string>]Specify SMBIOS type 1 fields.--smp <integer> (1 - N) (default = 1)The number of CPUs. Please use option -sockets instead.--sockets <integer> (1 - N) (default = 1)The number of CPU sockets.--sshkeys <filepath>cloud-init: Setup public SSH keys (one key per line, OpenSSH format). Proxmox VE Administration Guide 260 / 328--startdate (now | YYYY-MM-DD | YYYY-MM-DDTHH:MM:SS) (default = now)Set the initial date of the real time clock. Valid format for date are: now or 2006-06-17T16:01:21 or2006-06-17.--startup `[[order=]\d+] [,up=\d+] [,down=\d+] `Startup and shutdown behavior. Order is a non-negative number defining the general startup order.Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds,which specifies a delay to wait before the next VM is started or stopped.--tablet <boolean> (default = 1)Enable/disable the USB tablet device.--tdf <boolean> (default = 0)Enable/disable time drift fix.--template <boolean> (default = 0)Enable/disable Template.--unused[n] <string>Reference to unused volumes. This is used internally, and should not be modified manually.--usb[n] [host=]<HOSTUSBDEVICE|spice> [,usb3=<1|0>]Configure an USB device (n is 0 to 4).--vcpus <integer> (1 - N) (default = 0)Number of hotplugged vcpus.--vga <cirrus | qxl | qxl2 | qxl3 | qxl4 | serial0 | serial1 |serial2 | serial3 | std | vmware>Select the VGA type.--virtio[n] [file=]<volume> [,aio=<native|threads>] [,backup=<1|0>][,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>][,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,cyls=<integer>] [,detect_zeroes=<1|0>][,discard=<ignore|on>] [,format=<enum>] [,heads=<integer>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>][,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,iothread=<1|0>] [,mbps=<mbps>] [,mbps_max=<mbps>][,mbps_rd=<mbps>] [,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>][,mbps_wr_max=<mbps>] [,media=<cdrom|disk>] [,replicate=<1|0>][,rerror=<ignore|report|stop>] [,secs=<integer>] [,serial=<serial>][,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,trans=<none|lba|auto>] [,werror=<enum>]Use volume as VIRTIO hard disk (n is 0 to 15). Proxmox VE Administration Guide 261 / 328--vmstatestorage <string>Default storage for VM state volumes/files.--watchdog [[model=]<i6300esb|ib700>] [,action=<enum>]Create a virtual hardware watchdog device.qm showcmd <vmid> [OPTIONS]Show command line which is used to start the VM (debug info).<vmid>: <integer> (1 - N)The (unique) ID of the VM.--pretty <boolean> (default = 0)Puts each option on a new line to enhance human readabilityqm shutdown <vmid> [OPTIONS]Shutdown virtual machine. This is similar to pressing the power button on a physical machine.This will sendan ACPI event for the guest OS, which should then proceed to a clean shutdown.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--forceStop <boolean> (default = 0)Make sure the VM stops.--keepActive <boolean> (default = 0)Do not deactivate storage volumes.--skiplock <boolean>Ignore locks - only root is allowed to use this option.--timeout <integer> (0 - N)Wait maximal timeout seconds.qm snapshot <vmid> <snapname> [OPTIONS]Snapshot a VM.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<snapname>: <string>The name of the snapshot.--description <string>A textual description or comment. Proxmox VE Administration Guide 262 / 328--vmstate <boolean>Save the vmstateqm start <vmid> [OPTIONS]Start virtual machine.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--machine (pc|pc(-i440fx)?-\d+\.\d+(\.pxe)?|q35|pc-q35-\d+\.\d+(\.pxe)?)Specific the Qemu machine type.--migratedfrom <string>The cluster node name.--migration_network <string>CIDR of the (sub) network that is used for migration.--migration_type <insecure | secure>Migration traffic is encrypted using an SSH tunnel by default. On secure, completely private networksthis can be disabled to increase performance.--skiplock <boolean>Ignore locks - only root is allowed to use this option.--stateuri <string>Some command save/restore state from this location.--targetstorage <string>Target storage for the migration. (Can be 1 to use the same storage id as on the source node.)qm status <vmid> [OPTIONS]Show VM status.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--verbose <boolean>Verbose output formatqm stop <vmid> [OPTIONS]Stop virtual machine. The qemu process will exit immediately. Thisis akin to pulling the power plug of arunning computer and may damage the VM data Proxmox VE Administration Guide 263 / 328<vmid>: <integer> (1 - N)The (unique) ID of the VM.--keepActive <boolean> (default = 0)Do not deactivate storage volumes.--migratedfrom <string>The cluster node name.--skiplock <boolean>Ignore locks - only root is allowed to use this option.--timeout <integer> (0 - N)Wait maximal timeout seconds.qm suspend <vmid> [OPTIONS]Suspend virtual machine.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--skiplock <boolean>Ignore locks - only root is allowed to use this option.qm template <vmid> [OPTIONS]Create a Template.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--disk <efidisk0 | ide0 | ide1 | ide2 | ide3 | sata0 | sata1 |sata2 | sata3 | sata4 | sata5 | scsi0 | scsi1 | scsi10 | scsi11 |scsi12 | scsi13 | scsi2 | scsi3 | scsi4 | scsi5 | scsi6 | scsi7 |scsi8 | scsi9 | virtio0 | virtio1 | virtio10 | virtio11 | virtio12| virtio13 | virtio14 | virtio15 | virtio2 | virtio3 | virtio4 |virtio5 | virtio6 | virtio7 | virtio8 | virtio9>If you want to convert only 1 disk to base image.qm terminal <vmid> [OPTIONS]Open a terminal using a serial device (The VM need to have a serial device configured, for example serial0:socket)<vmid>: <integer> (1 - N)The (unique) ID of the VM. Proxmox VE Administration Guide 264 / 328--escape <string> (default = ˆO)Escape character.--iface <serial0 | serial1 | serial2 | serial3>Select the serial device. By default we simply use the first suitable device.qm unlink <vmid> --idlist <string> [OPTIONS]Unlink/delete disk images.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--force <boolean>Force physical removal. Without this, we simple remove the disk from the config file and create anadditional configuration entry called unused[n], which contains the volume ID. Unlink of unused[n]always cause physical removal.--idlist <string>A list of disk IDs you want to delete.qm unlock <vmid>Unlock the VM.<vmid>: <integer> (1 - N)The (unique) ID of the VM.qm vncproxy <vmid>Proxy VM VNC traffic to stdin/stdout<vmid>: <integer> (1 - N)The (unique) ID of the VM.qm wait <vmid> [OPTIONS]Wait until the VM is stopped.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--timeout <integer> (1 - N)Timeout in seconds. Default is to wait forever. Proxmox VE Administration Guide 265 / 328A.7 qmrestore - Restore QemuServer vzdump Backupsqmrestore helpqmrestore <archive> <vmid> [OPTIONS]Restore QemuServer vzdump backups.<archive>: <string>The backup file. You can pass - to read from standard input.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--bwlimit <number> (0 - N)Override i/o bandwidth limit (in KiB/s).--force <boolean>Allow to overwrite existing VM.--pool <string>Add the VM to the specified pool.--storage <string>Default storage.--unique <boolean>Assign a unique random ethernet address.A.8 pct - Proxmox Container Toolkitpct <COMMAND> [ARGS] [OPTIONS]pct clone <vmid> <newid> [OPTIONS]Create a container clone/copy<vmid>: <integer> (1 - N)The (unique) ID of the VM.<newid>: <integer> (1 - N)VMID for the clone.--description <string>Description for the new CT.--full <boolean>Create a full copy of all disks. This is always done when you clone a normal CT. For CT templates, wetry to create a linked clone by default. Proxmox VE Administration Guide 266 / 328--hostname <string>Set a hostname for the new CT.--pool <string>Add the new CT to the specified pool.--snapname <string>The name of the snapshot.--storage <string>Target storage for full clone.--target <string>Target node. Only allowed if the original VM is on shared storage.pct config <vmid>Get container configuration.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct console <vmid>Launch a console for the specified container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct cpusetsPrint the list of assigned CPU sets.pct create <vmid> <ostemplate> [OPTIONS]Create or restore a container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<ostemplate>: <string>The OS template or backup file.--arch <amd64 | i386> (default = amd64)OS architecture type.--bwlimit <number> (0 - N)Override i/o bandwidth limit (in KiB/s). Proxmox VE Administration Guide 15 / 328NoteIf set to 0, no swap volume will be created.maxrootDefines the maximum size of the root volume, which stores the operation system. The maximumlimit of the root volume size is hdsize/4.maxvzDefines the maximum size of the data volume. The actual size of the data volume is:datasize = hdsize - rootsize - swapsize - minfreeWhere datasize cannot be bigger than maxvz.NoteIn case of LVM thin, the data pool will only be created if datasize is bigger than 4GB.NoteIf set to 0, no data volume will be created and the storage configuration will be adapted accordingly.minfreeDefines the amount of free space left in LVM volume group pve. With more than 128GB storageavailable the default is 16GB, else hdsize/8 will be used.NoteLVM requires free space in the VG for snapshot creation (not required for lvmthin snapshots).2.2.2 ZFS Performance TipsZFS uses a lot of memory, so it is best to add additional RAM if you want to use ZFS. A good calculation is4GB plus 1GB RAM for each TB RAW disk space.ZFS also provides the feature to use a fast SSD drive as write cache. The write cache is called the ZFSIntent Log (ZIL). You can add that after installation using the following command:zpool add <pool-name> log </dev/path_to_fast_ssd>2.3 Install Proxmox VE on DebianProxmox VE ships as a set of Debian packages, so you can install it on top of a normal Debian installation.After configuring the repositories, you need to run:apt-get updateapt-get install proxmox-ve Proxmox VE Administration Guide 267 / 328--cmode <console | shell | tty> (default = tty)Console mode. By default, the console command tries to open a connection to one of the availabletty devices. By setting cmode to console it tries to attach to /dev/console instead. If you set cmode toshell, it simply invokes a shell inside the container (no login).--console <boolean> (default = 1)Attach a console device (/dev/console) to the container.--cores <integer> (1 - 128)The number of cores assigned to the container. A container can use all available cores by default.--cpulimit <number> (0 - 128) (default = 0)Limit of CPU usage.NoteIf the computer has 2 CPUs, it has a total of 2 CPU time. Value 0 indicates no CPU limit.--cpuunits <integer> (0 - 500000) (default = 1024)CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, themore CPU time this VM gets. Number is relative to the weights of all the other running VMs.NoteYou can disable fair-scheduler configuration by setting this to 0.--description <string>Container description. Only used on the configuration web interface.--force <boolean>Allow to overwrite existing container.--hostname <string>Set a host name for the container.--ignore-unpack-errors <boolean>Ignore errors when extracting the template.--lock <backup | migrate | rollback | snapshot>Lock/unlock the VM.--memory <integer> (16 - N) (default = 512)Amount of RAM for the VM in MB. Proxmox VE Administration Guide 268 / 328--mp[n] [volume=]<volume> ,mp=<Path> [,acl=<1|0>] [,backup=<1|0>][,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>][,size=<DiskSize>]Use volume as container mount point.--nameserver <string>Sets DNS server IP address for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.--net[n] name=<string> [,bridge=<bridge>] [,firewall=<1|0>] [,gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,hwaddr=<XX:XX:XX:XX:XX:XX>][,ip=<(IPv4/CIDR|dhcp|manual)>] [,ip6=<(IPv6/CIDR|auto|dhcp|manual)>] [,mtu=<integer>] [,rate=<mbps>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,type=<veth>]Specifies network interfaces for the container.--onboot <boolean> (default = 0)Specifies whether a VM will be started during system bootup.--ostype <alpine | archlinux | centos | debian | fedora | gentoo |opensuse | ubuntu | unmanaged>OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scriptsin /usr/share/lxc/config/<ostype>.common.conf. Value unmanaged can be used to skip and OS spe-cific setup.--passwordSets root password inside container.--pool <string>Add the VM to the specified pool.--protection <boolean> (default = 0)Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.--restore <boolean>Mark this as restore task.--rootfs [volume=]<volume> [,acl=<1|0>] [,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>] [,size=<DiskSize>]Use volume as container root.--searchdomain <string>Sets DNS search domains for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.--ssh-public-keys <filepath>Setup public SSH keys (one key per line, OpenSSH format). Proxmox VE Administration Guide 269 / 328--startup `[[order=]\d+] [,up=\d+] [,down=\d+] `Startup and shutdown behavior. Order is a non-negative number defining the general startup order.Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds,which specifies a delay to wait before the next VM is started or stopped.--storage <string> (default = local)Default Storage.--swap <integer> (0 - N) (default = 512)Amount of SWAP for the VM in MB.--template <boolean> (default = 0)Enable/disable Template.--tty <integer> (0 - 6) (default = 2)Specify the number of tty available to the container--unprivileged <boolean> (default = 0)Makes the container run as unprivileged user. (Should not be modified manually.)--unused[n] <string>Reference to unused volumes. This is used internally, and should not be modified manually.pct delsnapshot <vmid> <snapname> [OPTIONS]Delete a LXC snapshot.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<snapname>: <string>The name of the snapshot.--force <boolean>For removal from config file, even if removing disk snapshots fails.pct destroy <vmid>Destroy the container (also delete all uses files).<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct df <vmid>Get the container’s current disk usage. Proxmox VE Administration Guide 270 / 328<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct enter <vmid>Launch a shell for the specified container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct exec <vmid> [<extra-args>]Launch a command inside the specified container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<extra-args>: <array>Extra arguments as arraypct fsck <vmid> [OPTIONS]Run a filesystem check (fsck) on a container volume.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--device <mp0 | mp1 | mp2 | mp3 | mp4 | mp5 | mp6 | mp7 | mp8 | mp9| rootfs>A volume on which to run the filesystem check--force <boolean> (default = 0)Force checking, even if the filesystem seems cleanpct help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pct listLXC container index (per node).pct listsnapshot <vmid>List all snapshots. Proxmox VE Administration Guide 271 / 328<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct migrate <vmid> <target> [OPTIONS]Migrate the container to another node. Creates a new migration task.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<target>: <string>Target node.--force <boolean>Force migration despite local bind / device mounts. NOTE: deprecated, use shared property of mountpoint instead.--online <boolean>Use online/live migration.--restart <boolean>Use restart migration--timeout <integer> (default = 180)Timeout in seconds for shutdown for restart migrationpct mount <vmid>Mount the container’s filesystem on the host. This will hold a lock on the container and is meant for emer-gency maintenance only as it will prevent further operations on the container other than start and stop.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct move_volume <vmid> <volume> <storage> [OPTIONS]Move a rootfs-/mp-volume to a different storage<vmid>: <integer> (1 - N)The (unique) ID of the VM.<volume>: <mp0 | mp1 | mp2 | mp3 | mp4 | mp5 | mp6 | mp7 | mp8 | mp9| rootfs>Volume which will be moved.<storage>: <string>Target Storage. Proxmox VE Administration Guide 272 / 328--delete <boolean> (default = 0)Delete the original volume after successful copy. By default the original is kept as an unused volumeentry.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.pct pull <vmid> <path> <destination> [OPTIONS]Copy a file from the container to the local system.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<path>: <string>Path to a file inside the container to pull.<destination>: <string>Destination--group <string>Owner group name or id.--perms <string>File permissions to use (octal by default, prefix with 0x for hexadecimal).--user <string>Owner user name or id.pct push <vmid> <file> <destination> [OPTIONS]Copy a local file to the container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<file>: <string>Path to a local file.<destination>: <string>Destination inside the container to write to.--group <string>Owner group name or id. When using a name it must exist inside the container.--perms <string>File permissions to use (octal by default, prefix with 0x for hexadecimal). Proxmox VE Administration Guide 273 / 328--user <string>Owner user name or id. When using a name it must exist inside the container.pct resize <vmid> <disk> <size> [OPTIONS]Resize a container mount point.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<disk>: <mp0 | mp1 | mp2 | mp3 | mp4 | mp5 | mp6 | mp7 | mp8 | mp9 |rootfs>The disk you want to resize.<size>: \+?\d+(\.\d+)?[KMGT]?The new size. With the + sign the value is added to the actual size of the volume and without it, thevalue is taken as an absolute one. Shrinking disk size is not supported.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.pct restore <vmid> <ostemplate> [OPTIONS]Create or restore a container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<ostemplate>: <string>The OS template or backup file.--arch <amd64 | i386> (default = amd64)OS architecture type.--bwlimit <number> (0 - N)Override i/o bandwidth limit (in KiB/s).--cmode <console | shell | tty> (default = tty)Console mode. By default, the console command tries to open a connection to one of the availabletty devices. By setting cmode to console it tries to attach to /dev/console instead. If you set cmode toshell, it simply invokes a shell inside the container (no login).--console <boolean> (default = 1)Attach a console device (/dev/console) to the container.--cores <integer> (1 - 128)The number of cores assigned to the container. A container can use all available cores by default. Proxmox VE Administration Guide 274 / 328--cpulimit <number> (0 - 128) (default = 0)Limit of CPU usage.NoteIf the computer has 2 CPUs, it has a total of 2 CPU time. Value 0 indicates no CPU limit.--cpuunits <integer> (0 - 500000) (default = 1024)CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, themore CPU time this VM gets. Number is relative to the weights of all the other running VMs.NoteYou can disable fair-scheduler configuration by setting this to 0.--description <string>Container description. Only used on the configuration web interface.--force <boolean>Allow to overwrite existing container.--hostname <string>Set a host name for the container.--ignore-unpack-errors <boolean>Ignore errors when extracting the template.--lock <backup | migrate | rollback | snapshot>Lock/unlock the VM.--memory <integer> (16 - N) (default = 512)Amount of RAM for the VM in MB.--mp[n] [volume=]<volume> ,mp=<Path> [,acl=<1|0>] [,backup=<1|0>][,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>][,size=<DiskSize>]Use volume as container mount point.--nameserver <string>Sets DNS server IP address for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.--net[n] name=<string> [,bridge=<bridge>] [,firewall=<1|0>] [,gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,hwaddr=<XX:XX:XX:XX:XX:XX>][,ip=<(IPv4/CIDR|dhcp|manual)>] [,ip6=<(IPv6/CIDR|auto|dhcp|manual)>] [,mtu=<integer>] [,rate=<mbps>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,type=<veth>]Specifies network interfaces for the container. Proxmox VE Administration Guide 275 / 328--onboot <boolean> (default = 0)Specifies whether a VM will be started during system bootup.--ostype <alpine | archlinux | centos | debian | fedora | gentoo |opensuse | ubuntu | unmanaged>OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scriptsin /usr/share/lxc/config/<ostype>.common.conf. Value unmanaged can be used to skip and OS spe-cific setup.--passwordSets root password inside container.--pool <string>Add the VM to the specified pool.--protection <boolean> (default = 0)Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.--rootfs [volume=]<volume> [,acl=<1|0>] [,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>] [,size=<DiskSize>]Use volume as container root.--searchdomain <string>Sets DNS search domains for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.--ssh-public-keys <filepath>Setup public SSH keys (one key per line, OpenSSH format).--startup `[[order=]\d+] [,up=\d+] [,down=\d+] `Startup and shutdown behavior. Order is a non-negative number defining the general startup order.Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds,which specifies a delay to wait before the next VM is started or stopped.--storage <string> (default = local)Default Storage.--swap <integer> (0 - N) (default = 512)Amount of SWAP for the VM in MB.--template <boolean> (default = 0)Enable/disable Template.--tty <integer> (0 - 6) (default = 2)Specify the number of tty available to the container Proxmox VE Administration Guide 276 / 328--unprivileged <boolean> (default = 0)Makes the container run as unprivileged user. (Should not be modified manually.)--unused[n] <string>Reference to unused volumes. This is used internally, and should not be modified manually.pct resume <vmid>Resume the container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct rollback <vmid> <snapname>Rollback LXC state to specified snapshot.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<snapname>: <string>The name of the snapshot.pct set <vmid> [OPTIONS]Set container options.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--arch <amd64 | i386> (default = amd64)OS architecture type.--cmode <console | shell | tty> (default = tty)Console mode. By default, the console command tries to open a connection to one of the availabletty devices. By setting cmode to console it tries to attach to /dev/console instead. If you set cmode toshell, it simply invokes a shell inside the container (no login).--console <boolean> (default = 1)Attach a console device (/dev/console) to the container.--cores <integer> (1 - 128)The number of cores assigned to the container. A container can use all available cores by default.--cpulimit <number> (0 - 128) (default = 0)Limit of CPU usage. Proxmox VE Administration Guide 16 / 328Installing on top of an existing Debian installation looks easy, but it presumes that you have correctly installedthe base system, and you know how you want to configure and use the local storage. Network configurationis also completely up to you.In general, this is not trivial, especially when you use LVM or ZFS.You can find a detailed step by step howto on the wiki.2.4 Install from USB StickThe Proxmox VE installation media is now a hybrid ISO image, working in two ways:• An ISO image file ready to burn on CD• A raw sector (IMG) image file ready to directly copy to flash media (USB Stick)Using USB sticks is faster and more environmental friendly and therefore the recommended way to installProxmox VE.2.4.1 Prepare a USB flash drive as install mediumIn order to boot the installation media, copy the ISO image to a USB media.First download the ISO image from https://www.proxmox.com/en/downloads/category/iso-images-pveYou need at least a 1 GB USB media.NoteUsing UNetbootin or Rufus does not work.ImportantMake sure that the USB media is not mounted and does not contain any important data.2.4.2 Instructions for GNU/LinuxYou can simply use dd on UNIX like systems. First download the ISO image, then plug in the USB stick. Youneed to find out what device name gets assigned to the USB stick (see below). Then run:dd if=proxmox-ve_*.iso of=/dev/XYZ bs=1MNoteBe sure to replace /dev/XYZ with the correct device name.CautionBe very careful, and do not overwrite the hard disk! Proxmox VE Administration Guide 277 / 328NoteIf the computer has 2 CPUs, it has a total of 2 CPU time. Value 0 indicates no CPU limit.--cpuunits <integer> (0 - 500000) (default = 1024)CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, themore CPU time this VM gets. Number is relative to the weights of all the other running VMs.NoteYou can disable fair-scheduler configuration by setting this to 0.--delete <string>A list of settings you want to delete.--description <string>Container description. Only used on the configuration web interface.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--hostname <string>Set a host name for the container.--lock <backup | migrate | rollback | snapshot>Lock/unlock the VM.--memory <integer> (16 - N) (default = 512)Amount of RAM for the VM in MB.--mp[n] [volume=]<volume> ,mp=<Path> [,acl=<1|0>] [,backup=<1|0>][,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>][,size=<DiskSize>]Use volume as container mount point.--nameserver <string>Sets DNS server IP address for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.--net[n] name=<string> [,bridge=<bridge>] [,firewall=<1|0>] [,gw=<GatewayIPv4>] [,gw6=<GatewayIPv6>] [,hwaddr=<XX:XX:XX:XX:XX:XX>][,ip=<(IPv4/CIDR|dhcp|manual)>] [,ip6=<(IPv6/CIDR|auto|dhcp|manual)>] [,mtu=<integer>] [,rate=<mbps>] [,tag=<integer>] [,trunks=<vlanid[;vlanid...]>] [,type=<veth>]Specifies network interfaces for the container. Proxmox VE Administration Guide 278 / 328--onboot <boolean> (default = 0)Specifies whether a VM will be started during system bootup.--ostype <alpine | archlinux | centos | debian | fedora | gentoo |opensuse | ubuntu | unmanaged>OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scriptsin /usr/share/lxc/config/<ostype>.common.conf. Value unmanaged can be used to skip and OS spe-cific setup.--protection <boolean> (default = 0)Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.--rootfs [volume=]<volume> [,acl=<1|0>] [,quota=<1|0>] [,replicate=<1|0>] [,ro=<1|0>] [,shared=<1|0>] [,size=<DiskSize>]Use volume as container root.--searchdomain <string>Sets DNS search domains for a container. Create will automatically use the setting from the host ifyou neither set searchdomain nor nameserver.--startup `[[order=]\d+] [,up=\d+] [,down=\d+] `Startup and shutdown behavior. Order is a non-negative number defining the general startup order.Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds,which specifies a delay to wait before the next VM is started or stopped.--swap <integer> (0 - N) (default = 512)Amount of SWAP for the VM in MB.--template <boolean> (default = 0)Enable/disable Template.--tty <integer> (0 - 6) (default = 2)Specify the number of tty available to the container--unprivileged <boolean> (default = 0)Makes the container run as unprivileged user. (Should not be modified manually.)--unused[n] <string>Reference to unused volumes. This is used internally, and should not be modified manually.pct shutdown <vmid> [OPTIONS]Shutdown the container. This will trigger a clean shutdown of the container, see lxc-stop(1) for details.<vmid>: <integer> (1 - N)The (unique) ID of the VM. Proxmox VE Administration Guide 279 / 328--forceStop <boolean> (default = 0)Make sure the Container stops.--timeout <integer> (0 - N) (default = 60)Wait maximal timeout seconds.pct snapshot <vmid> <snapname> [OPTIONS]Snapshot a container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<snapname>: <string>The name of the snapshot.--description <string>A textual description or comment.pct start <vmid> [OPTIONS]Start the container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--skiplock <boolean>Ignore locks - only root is allowed to use this option.pct status <vmid> [OPTIONS]Show CT status.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--verbose <boolean>Verbose output formatpct stop <vmid> [OPTIONS]Stop the container. This will abruptly stop all processes running in the container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.--skiplock <boolean>Ignore locks - only root is allowed to use this option. Proxmox VE Administration Guide 280 / 328pct suspend <vmid>Suspend the container.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct template <vmid>Create a Template.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct unlock <vmid>Unlock the VM.<vmid>: <integer> (1 - N)The (unique) ID of the VM.pct unmount <vmid>Unmount the container’s filesystem.<vmid>: <integer> (1 - N)The (unique) ID of the VM.A.9 pveam - Proxmox VE Appliance Managerpveam <COMMAND> [ARGS] [OPTIONS]pveam available [OPTIONS]List available templates.--section <system | turnkeylinux>Restrict list to specified section.pveam download <storage> <template>Download appliance templates.<storage>: <string>The storage where the template will be stored<template>: <string>The template wich will downloaded Proxmox VE Administration Guide 281 / 328pveam help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pveam list <storage>Get list of all templates on storage<storage>: <string>Only list templates on specified storagepveam remove <template_path>Remove a template.<template_path>: <string>The template to remove.pveam updateUpdate Container Template Database.A.10 pvecm - Proxmox VE Cluster Managerpvecm <COMMAND> [ARGS] [OPTIONS]pvecm add <hostname> [OPTIONS]Adds the current node to an existing cluster.<hostname>: <string>Hostname (or IP) of an existing cluster member.--fingerprint ([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}Certificate SHA 256 fingerprint.--force <boolean>Do not throw error if node already exists.--nodeid <integer> (1 - N)Node id for this node.--ring0_addr <string> (default = Hostname of the node)Hostname (or IP) of the corosync ring0 address of this node. Proxmox VE Administration Guide 282 / 328--ring1_addr <string>Hostname (or IP) of the corosync ring1 address of this node. Requires a valid configured ring 1(bindnet1_addr) in the cluster.--use_ssh <boolean>Always use SSH to join, even if peer may do it over API.--votes <integer> (0 - N)Number of votes for this nodepvecm addnode <node> [OPTIONS]Adds a node to the cluster configuration.<node>: <string>The cluster node name.--force <boolean>Do not throw error if node already exists.--nodeid <integer> (1 - N)Node id for this node.--ring0_addr <string> (default = Hostname of the node)Hostname (or IP) of the corosync ring0 address of this node.--ring1_addr <string>Hostname (or IP) of the corosync ring1 address of this node. Requires a valid configured ring 1(bindnet1_addr) in the cluster.--votes <integer> (0 - N)Number of votes for this nodepvecm create <clustername> [OPTIONS]Generate new cluster configuration.<clustername>: <string>The name of the cluster.--bindnet0_addr <string>This specifies the network address the corosync ring 0 executive should bind to and defaults to thelocal IP address of the node.--bindnet1_addr <string>This specifies the network address the corosync ring 1 executive should bind to and is optional. Proxmox VE Administration Guide 283 / 328--nodeid <integer> (1 - N)Node id for this node.--ring0_addr <string> (default = Hostname of the node)Hostname (or IP) of the corosync ring0 address of this node.--ring1_addr <string>Hostname (or IP) of the corosync ring1 address of this node. Requires a valid configured ring 1(bindnet1_addr) in the cluster.--votes <integer> (1 - N)Number of votes for this node.pvecm delnode <node>Removes a node from the cluster configuration.<node>: <string>The cluster node name.pvecm expected <expected>Tells corosync a new value of expected votes.<expected>: <integer> (1 - N)Expected votespvecm help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pvecm keygen <filename>Generate new cryptographic key for corosync.<filename>: <string>Output file namepvecm mtunnel [<extra-args>] [OPTIONS]Used by VM/CT migration - do not use manually.<extra-args>: <array>Extra arguments as array Proxmox VE Administration Guide 284 / 328--get_migration_ip <boolean> (default = 0)return the migration IP, if configured--migration_network <string>the migration network used to detect the local migration IP--run-command <boolean>Run a command with a tcp socket as standard input. The IP address and port are printed via thiscommand’s stdandard output first, each on a separate line.pvecm nodesDisplays the local view of the cluster nodes.pvecm statusDisplays the local view of the cluster status.pvecm updatecerts [OPTIONS]Update node certificates (and generate all needed files/directories).--force <boolean>Force generation of new SSL certifate.--silent <boolean>Ignore errors (i.e. when cluster has no quorum).A.11 pvesr - Proxmox VE Storage Replicationpvesr <COMMAND> [ARGS] [OPTIONS]pvesr create-local-job <id> <target> [OPTIONS]Create a new replication job<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.<target>: <string>Target node.--comment <string>Description.--disable <boolean>Flag to disable/deactivate the entry.--rate <number> (1 - N)Rate limit in mbps (megabytes per second) as floating point number. Proxmox VE Administration Guide 285 / 328--remove_job <full | local>Mark the replication job for removal. The job will remove all local replication snapshots. When setto full, it also tries to remove replicated volumes on the target. The job then removes itself from theconfiguration file.--schedule <string> (default =*/15)Storage replication schedule. The format is a subset of systemd calender events.--source <string>Source of the replication.pvesr delete <id> [OPTIONS]Mark replication job for removal.<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.--force <boolean> (default = 0)Will remove the jobconfig entry, but will not cleanup.--keep <boolean> (default = 0)Keep replicated data at target (do not remove).pvesr disable <id>Disable a replication job.<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.pvesr enable <id>Enable a replication job.<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.pvesr finalize-local-job <id> [<extra-args>] [OPTIONS]Finalize a replication job. This removes all replications snapshots with timestamps different than <last_sync>.<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>. Proxmox VE Administration Guide 286 / 328<extra-args>: <array>The list of volume IDs to consider.--last_sync <integer> (0 - N)Time (UNIX epoch) of last successful sync. If not specified, all replication snapshots gets removed.pvesr help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pvesr listList replication jobs.pvesr prepare-local-job <id> [<extra-args>] [OPTIONS]Prepare for starting a replication job. This is called on the target node before replication starts. This callis for internal use, and return a JSON object on stdout. The method first test if VM <vmid> reside on thelocal node. If so, stop immediately. After that the method scans all volume IDs for snapshots, and removesall replications snapshots with timestamps different than <last_sync>. It also removes any unused volumes.Returns a hash with boolean markers for all volumes with existing replication snapshots.<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.<extra-args>: <array>The list of volume IDs to consider.--force <boolean> (default = 0)Allow to remove all existion volumes (empty volume list).--last_sync <integer> (0 - N)Time (UNIX epoch) of last successful sync. If not specified, all replication snapshots get removed.--parent_snapname <string>The name of the snapshot.--scan <string>List of storage IDs to scan for stale volumes.pvesr read <id>Read replication job configuration. Proxmox VE Administration Guide iiiContents1 Introduction 11.1 Central Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Flexible Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Integrated Backup and Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 High Availability Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.5 Flexible Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.6 Integrated Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.7 Why Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.8 Your benefit with Proxmox VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.9 Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.9.1 Proxmox VE Wiki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.9.2 Community Support Forum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.9.3 Mailing Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.9.4 Commercial Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.9.5 Bug Tracker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.10 Project History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.11 Improving the Proxmox VE Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Installing Proxmox VE 72.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1.1 Minimum Requirements, for Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1.2 Recommended System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 82.1.3 Simple Performance Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.1.4 Supported web browsers for accessing the web interface . . . . . . . . . . . . . . . . 82.2 Using the Proxmox VE Installation CD-ROM . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2.1 Advanced LVM Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2.2 ZFS Performance Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.3 Install Proxmox VE on Debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Proxmox VE Administration Guide 17 / 328Find Correct USB Device NameYou can compare the last lines of dmesg command before and after the insertion, or use the lsblk command.Open a terminal and run:lsblkThen plug in your USB media and run the command again:lsblkA new device will appear, and this is the USB device you want to use.2.4.3 Instructions for OSXOpen the terminal (query Terminal in Spotlight).Convert the .iso file to .img using the convert option of hdiutil for example.hdiutil convert -format UDRW -o proxmox-ve_*.dmg proxmox-ve_*.isoTipOS X tends to put the .dmg ending on the output file automatically.To get the current list of devices run the command again:diskutil listNow insert your USB flash media and run this command again to determine the device node assigned toyour flash media (e.g. /dev/diskX).diskutil listdiskutil unmountDisk /dev/diskXNotereplace X with the disk number from the last command.sudo dd if=proxmox-ve_*.dmg of=/dev/rdiskN bs=1m2.4.4 Instructions for WindowsDownload Etcher from https://etcher.io , select the ISO and your USB Drive.If this doesn’t work, alternatively use the OSForensics USB installer from http://www.osforensics.com/portability.html2.4.5 Boot your server from USB mediaConnect your USB media to your server and make sure that the server boots from USB (see server BIOS).Then follow the installation wizard. Proxmox VE Administration Guide 287 / 328<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.pvesr run [OPTIONS]This method is called by the systemd-timer and executes all (or a specific) sync jobs.--id [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.--mail <boolean> (default = 0)Send an email notification in case of a failure.--verbose <boolean> (default = 0)Print more verbose logs to stdout.pvesr schedule-now <id>Schedule replication job to start as soon as possible.<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.pvesr set-state <vmid> <state>Set the job replication state on migration. This call is for internal use. It will accept the job state as ja JSONobj.<vmid>: <integer> (1 - N)The (unique) ID of the VM.<state>: <string>Job state as JSON decoded string.pvesr status [OPTIONS]List status of all replication jobs on this node.--guest <integer> (1 - N)Only list replication jobs for this guest.pvesr update <id> [OPTIONS]Update replication job configuration. Proxmox VE Administration Guide 288 / 328<id>: [1-9][0-9]{2,8}-\d{1,9}Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e.<GUEST>-<JOBNUM>.--comment <string>Description.--delete <string>A list of settings you want to delete.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--disable <boolean>Flag to disable/deactivate the entry.--rate <number> (1 - N)Rate limit in mbps (megabytes per second) as floating point number.--remove_job <full | local>Mark the replication job for removal. The job will remove all local replication snapshots. When setto full, it also tries to remove replicated volumes on the target. The job then removes itself from theconfiguration file.--schedule <string> (default =*/15)Storage replication schedule. The format is a subset of systemd calender events.--source <string>Source of the replication.A.12 pveum - Proxmox VE User Managerpveum <COMMAND> [ARGS] [OPTIONS]pveum acldel <path> --roles <string> [OPTIONS]Update Access Control List (add or remove permissions).<path>: <string>Access control path--groups <string>List of groups.--propagate <boolean> (default = 1)Allow to propagate (inherit) permissions. Proxmox VE Administration Guide 289 / 328--roles <string>List of roles.--users <string>List of users.pveum aclmod <path> --roles <string> [OPTIONS]Update Access Control List (add or remove permissions).<path>: <string>Access control path--groups <string>List of groups.--propagate <boolean> (default = 1)Allow to propagate (inherit) permissions.--roles <string>List of roles.--users <string>List of users.pveum groupadd <groupid> [OPTIONS]Create new group.<groupid>: <string>no description available--comment <string>no description availablepveum groupdel <groupid>Delete group.<groupid>: <string>no description availablepveum groupmod <groupid> [OPTIONS]Update group data.<groupid>: <string>no description available Proxmox VE Administration Guide 290 / 328--comment <string>no description availablepveum help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pveum passwd <userid>Change user password.<userid>: <string>User IDpveum roleadd <roleid> [OPTIONS]Create new role.<roleid>: <string>no description available--privs <string>no description availablepveum roledel <roleid>Delete role.<roleid>: <string>no description availablepveum rolemod <roleid> --privs <string> [OPTIONS]Create new role.<roleid>: <string>no description available--append <boolean>no description availableNoteRequires option(s): privs Proxmox VE Administration Guide 291 / 328--privs <string>no description availablepveum ticket <username> [OPTIONS]Create or verify authentication ticket.<username>: <string>User name--otp <string>One-time password for Two-factor authentication.--path <string>Verify ticket, and check if user have access privs on pathNoteRequires option(s): privs--privs <string>Verify ticket, and check if user have access privs on pathNoteRequires option(s): path--realm <string>You can optionally pass the realm using this parameter. Normally the realm is simply added to theusername <username>@<relam>.pveum useradd <userid> [OPTIONS]Create new user.<userid>: <string>User ID--comment <string>no description available--email <string>no description available--enable <boolean> (default = 1)Enable the account (default). You can set this to 0 to disable the accout Proxmox VE Administration Guide 292 / 328--expire <integer> (0 - N)Account expiration date (seconds since epoch). 0 means no expiration date.--firstname <string>no description available--groups <string>no description available--keys <string>Keys for two factor auth (yubico).--lastname <string>no description available--passwordInitial password.pveum userdel <userid>Delete user.<userid>: <string>User IDpveum usermod <userid> [OPTIONS]Update user configuration.<userid>: <string>User ID--append <boolean>no description availableNoteRequires option(s): groups--comment <string>no description available--email <string>no description available--enable <boolean>Enable/disable the account. Proxmox VE Administration Guide 293 / 328--expire <integer> (0 - N)Account expiration date (seconds since epoch). 0 means no expiration date.--firstname <string>no description available--groups <string>no description available--keys <string>Keys for two factor auth (yubico).--lastname <string>no description availableA.13 vzdump - Backup Utility for VMs and Containersvzdump helpvzdump {<vmid>} [OPTIONS]Create backup.<vmid>: <string>The ID of the guest system you want to backup.--all <boolean> (default = 0)Backup all known guest systems on this host.--bwlimit <integer> (0 - N) (default = 0)Limit I/O bandwidth (KBytes per second).--compress <0 | 1 | gzip | lzo> (default = 0)Compress dump file.--dumpdir <string>Store resulting files to specified directory.--exclude <string>Exclude specified guest systems (assumes --all)--exclude-path <string>Exclude certain files/directories (shell globs).--ionice <integer> (0 - 8) (default = 7)Set CFQ ionice priority. Proxmox VE Administration Guide 294 / 328--lockwait <integer> (0 - N) (default = 180)Maximal time to wait for the global lock (minutes).--mailnotification <always | failure> (default = always)Specify when to send an email--mailto <string>Comma-separated list of email addresses that should receive email notifications.--maxfiles <integer> (1 - N) (default = 1)Maximal number of backup files per guest system.--mode <snapshot | stop | suspend> (default = snapshot)Backup mode.--node <string>Only run if executed on this node.--pigz <integer> (default = 0)Use pigz instead of gzip when N>0. N=1 uses half of cores, N>1 uses N as thread count.--quiet <boolean> (default = 0)Be quiet.--remove <boolean> (default = 1)Remove old backup files if there are more than maxfiles backup files.--script <string>Use specified hook script.--size <integer> (500 - N) (default = 1024)Unused, will be removed in a future release.--stdexcludes <boolean> (default = 1)Exclude temporary files and logs.--stdout <boolean>Write tar to stdout, not to a file.--stop <boolean> (default = 0)Stop runnig backup jobs on this host.--stopwait <integer> (0 - N) (default = 10)Maximal time to wait until a guest system is stopped (minutes). Proxmox VE Administration Guide 295 / 328--storage <string>Store resulting file to this storage.--tmpdir <string>Store temporary files to specified directory.A.14 ha-manager - Proxmox VE HA Managerha-manager <COMMAND> [ARGS] [OPTIONS]ha-manager add <sid> [OPTIONS]Create a new HA resource.<sid>: <type>:<name>HA resource ID. This consists of a resource type followed by a resource specific name, separated withcolon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM orCT id as a shortcut (example: 100).--comment <string>Description.--group <string>The HA group identifier.--max_relocate <integer> (0 - N) (default = 1)Maximal number of service relocate tries when a service failes to start.--max_restart <integer> (0 - N) (default = 1)Maximal number of tries to restart the service on a node after its start failed.--state <disabled | enabled | ignored | started | stopped> (default =started)Requested resource state.--type <ct | vm>Resource type.ha-manager config [OPTIONS]List HA resources.--type <ct | vm>Only list resources of specific typeha-manager groupadd <group> --nodes <string> [OPTIONS]Create a new HA group. Proxmox VE Administration Guide 296 / 328<group>: <string>The HA group identifier.--comment <string>Description.--nodes <node>[:<pri>]{,<node>[:<pri>]}*List of cluster node names with optional priority.--nofailback <boolean> (default = 0)The CRM tries to run services on the node with the highest priority. If a node with higher priority comesonline, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.--restricted <boolean> (default = 0)Resources bound to restricted groups may only run on nodes defined by the group.--type <group>Group type.ha-manager groupconfigGet HA groups.ha-manager groupremove <group>Delete ha group configuration.<group>: <string>The HA group identifier.ha-manager groupset <group> [OPTIONS]Update ha group configuration.<group>: <string>The HA group identifier.--comment <string>Description.--delete <string>A list of settings you want to delete.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--nodes <node>[:<pri>]{,<node>[:<pri>]}*List of cluster node names with optional priority. Proxmox VE Administration Guide 18 / 328Chapter 3Host System AdministrationProxmox VE is based on the famous Debian Linux distribution. That means that you have access to thewhole world of Debian packages, and the base system is well documented. The Debian Administrator’sHandbook is available online, and provides a comprehensive introduction to the Debian operating system(see [Hertzog13]).A standard Proxmox VE installation uses the default repositories from Debian, so you get bug fixes andsecurity updates through that channel. In addition, we provide our own package repository to roll out allProxmox VE related packages. This includes updates to some Debian packages when necessary.We also deliver a specially optimized Linux kernel, where we enable all required virtualization and containerfeatures. That kernel includes drivers for ZFS, and several hardware drivers. For example, we ship Intelnetwork card drivers to support their newest hardware.The following sections will concentrate on virtualization related topics. They either explains things which aredifferent on Proxmox VE, or tasks which are commonly used on Proxmox VE. For other topics, please referto the standard Debian documentation.3.1 Package RepositoriesAll Debian based systems use APT as package management tool. The list of repositories is defined in /etc/apt/sources.list and .list files found inside /etc/apt/sources.d/. Updates can beinstalled directly using apt-get, or via the GUI.Apt sources.list files list one package repository per line, with the most preferred source listed first.Empty lines are ignored, and a # character anywhere on a line marks the remainder of that line as a com-ment. The information available from the configured sources is acquired by apt-get update.File /etc/apt/sources.listdeb http://ftp.debian.org/debian stretch main contrib# security updatesdeb http://security.debian.org stretch/updates main contribIn addition, Proxmox VE provides three different package repositories. Proxmox VE Administration Guide 297 / 328--nofailback <boolean> (default = 0)The CRM tries to run services on the node with the highest priority. If a node with higher priority comesonline, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.--restricted <boolean> (default = 0)Resources bound to restricted groups may only run on nodes defined by the group.ha-manager help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.ha-manager migrate <sid> <node>Request resource migration (online) to another node.<sid>: <type>:<name>HA resource ID. This consists of a resource type followed by a resource specific name, separated withcolon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM orCT id as a shortcut (example: 100).<node>: <string>The cluster node name.ha-manager relocate <sid> <node>Request resource relocatzion to another node. This stops the service on the old node, and restarts it on thetarget node.<sid>: <type>:<name>HA resource ID. This consists of a resource type followed by a resource specific name, separated withcolon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM orCT id as a shortcut (example: 100).<node>: <string>The cluster node name.ha-manager remove <sid>Delete resource configuration.<sid>: <type>:<name>HA resource ID. This consists of a resource type followed by a resource specific name, separated withcolon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM orCT id as a shortcut (example: 100). Proxmox VE Administration Guide 298 / 328ha-manager set <sid> [OPTIONS]Update resource configuration.<sid>: <type>:<name>HA resource ID. This consists of a resource type followed by a resource specific name, separated withcolon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM orCT id as a shortcut (example: 100).--comment <string>Description.--delete <string>A list of settings you want to delete.--digest <string>Prevent changes if current configuration file has different SHA1 digest. This can be used to preventconcurrent modifications.--group <string>The HA group identifier.--max_relocate <integer> (0 - N) (default = 1)Maximal number of service relocate tries when a service failes to start.--max_restart <integer> (0 - N) (default = 1)Maximal number of tries to restart the service on a node after its start failed.--state <disabled | enabled | ignored | started | stopped> (default =started)Requested resource state.ha-manager status [OPTIONS]Display HA manger status.--verbose <boolean> (default = 0)Verbose output. Include complete CRM and LRM status (JSON). Proxmox VE Administration Guide 299 / 328Appendix BService DaemonsB.1 pve-firewall - Proxmox VE Firewall Daemonpve-firewall <COMMAND> [ARGS] [OPTIONS]pve-firewall compileCompile and print firewall rules. This is useful for testing.pve-firewall help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pve-firewall localnetPrint information about local network.pve-firewall restartRestart the Proxmox VE firewall service.pve-firewall simulate [OPTIONS]Simulate firewall rules. This does not simulate kernel routing table. Instead, this simply assumes that routingfrom source zone to destination zone is possible.--dest <string>Destination IP address.--dport <integer>Destination port.--from (host|outside|vm\d+|ct\d+|vmbr\d+/\S+) (default = outside)Source zone. Proxmox VE Administration Guide 300 / 328--protocol (tcp|udp) (default = tcp)Protocol.--source <string>Source IP address.--sport <integer>Source port.--to (host|outside|vm\d+|ct\d+|vmbr\d+/\S+) (default = host)Destination zone.--verbose <boolean> (default = 0)Verbose output.pve-firewall start [OPTIONS]Start the Proxmox VE firewall service.--debug <boolean> (default = 0)Debug mode - stay in foregroundpve-firewall statusGet firewall status.pve-firewall stopStop firewall. This removes all Proxmox VE related iptable rules. The host is unprotected afterwards.B.2 pvedaemon - Proxmox VE API Daemonpvedaemon <COMMAND> [ARGS] [OPTIONS]pvedaemon help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pvedaemon restartRestart the daemon (or start if not running).pvedaemon start [OPTIONS]Start the daemon. Proxmox VE Administration Guide 301 / 328--debug <boolean> (default = 0)Debug mode - stay in foregroundpvedaemon statusGet daemon status.pvedaemon stopStop the daemon.B.3 pveproxy - Proxmox VE API Proxy Daemonpveproxy <COMMAND> [ARGS] [OPTIONS]pveproxy help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pveproxy restartRestart the daemon (or start if not running).pveproxy start [OPTIONS]Start the daemon.--debug <boolean> (default = 0)Debug mode - stay in foregroundpveproxy statusGet daemon status.pveproxy stopStop the daemon.B.4 pvestatd - Proxmox VE Status Daemonpvestatd <COMMAND> [ARGS] [OPTIONS]pvestatd help [OPTIONS]Get help about specified command. Proxmox VE Administration Guide 302 / 328--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pvestatd restartRestart the daemon (or start if not running).pvestatd start [OPTIONS]Start the daemon.--debug <boolean> (default = 0)Debug mode - stay in foregroundpvestatd statusGet daemon status.pvestatd stopStop the daemon.B.5 spiceproxy - SPICE Proxy Servicespiceproxy <COMMAND> [ARGS] [OPTIONS]spiceproxy help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.spiceproxy restartRestart the daemon (or start if not running).spiceproxy start [OPTIONS]Start the daemon.--debug <boolean> (default = 0)Debug mode - stay in foregroundspiceproxy statusGet daemon status.spiceproxy stopStop the daemon. Proxmox VE Administration Guide 303 / 328B.6 pmxcfs - Proxmox Cluster File Systempmxcfs [OPTIONS]Help Options:-h, --helpShow help optionsApplication Options:-d, --debugTurn on debug messages-f, --foregroundDo not daemonize server-l, --localForce local mode (ignore corosync.conf, force quorum)This service is usually started and managed using systemd toolset. The service is called pve-cluster.systemctl start pve-clustersystemctl stop pve-clustersystemctl status pve-clusterB.7 pve-ha-crm - Cluster Resource Manager Daemonpve-ha-crm <COMMAND> [ARGS] [OPTIONS]pve-ha-crm help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pve-ha-crm start [OPTIONS]Start the daemon.--debug <boolean> (default = 0)Debug mode - stay in foreground Proxmox VE Administration Guide 304 / 328pve-ha-crm statusGet daemon status.pve-ha-crm stopStop the daemon.B.8 pve-ha-lrm - Local Resource Manager Daemonpve-ha-lrm <COMMAND> [ARGS] [OPTIONS]pve-ha-lrm help [OPTIONS]Get help about specified command.--extra-args <array>Shows help for a specific command--verbose <boolean>Verbose output format.pve-ha-lrm start [OPTIONS]Start the daemon.--debug <boolean> (default = 0)Debug mode - stay in foregroundpve-ha-lrm statusGet daemon status.pve-ha-lrm stopStop the daemon. Proxmox VE Administration Guide 305 / 328Appendix CConfiguration FilesC.1 Datacenter ConfigurationThe file /etc/pve/datacenter.cfg is a configuration file for Proxmox VE. It contains cluster widedefault values used by all nodes.C.1.1 File FormatThe file uses a simple colon separated key/value format. Each line has the following format:OPTION: valueBlank lines in the file are ignored, and lines starting with a # character are treated as comments and are alsoignored.C.1.2 Optionsbwlimit: [clone=<LIMIT>] [,default=<LIMIT>] [,migration=<LIMIT>][,move=<LIMIT>] [,restore=<LIMIT>]Set bandwidth/io limits various operations.clone=<LIMIT>bandwidth limit in MiB/s for cloning disksdefault=<LIMIT>default bandwidth limit in MiB/smigration=<LIMIT>bandwidth limit in MiB/s for migrating guestsmove=<LIMIT>bandwidth limit in MiB/s for moving disksrestore=<LIMIT>bandwidth limit in MiB/s for restoring guests from backups Proxmox VE Administration Guide 306 / 328console: <applet | html5 | vv | xtermjs>Select the default Console viewer. You can either use the builtin java applet (VNC; deprecated andmaps to html5), an external virt-viewer comtatible application (SPICE), an HTML5 based vnc viewer(noVNC), or an HTML5 based console client (xtermjs). If the selected viewer is not available (e.g.SPICE not activated for the VM), the fallback is noVNC.email_from: <string>Specify email address to send notification from (default is root@$hostname)fencing: <both | hardware | watchdog> (default = watchdog)Set the fencing mode of the HA cluster. Hardware mode needs a valid configuration of fence devicesin /etc/pve/ha/fence.cfg. With both all two modes are used.Warninghardware and both are EXPERIMENTAL & WIPhttp_proxy: http://.*Specify external http proxy which is used for downloads (example: http://username:password@host:port/ )keyboard: <da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be |fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt |pt-br | sl | sv | tr>Default keybord layout for vnc server.language: <de | en>Default GUI language.mac_prefix: (?ˆi:[a-f0-9]{2}(?::[a-f0-9]{2}){0,2}:?)Prefix for autogenerated MAC addresses.max_workers: <integer> (1 - N)Defines how many workers (per node) are maximal started on actions like stopall VMs or task fromthe ha-manager.migration: [type=]<secure|insecure> [,network=<CIDR>]For cluster wide migration settings.network=<CIDR>CIDR of the (sub) network that is used for migration.type=<insecure | secure> (default = secure)Migration traffic is encrypted using an SSH tunnel by default. On secure, completely privatenetworks this can be disabled to increase performance. Proxmox VE Administration Guide 19 / 3283.1.1 Proxmox VE Enterprise RepositoryThis is the default, stable and recommended repository, available for all Proxmox VE subscription users. Itcontains the most stable packages, and is suitable for production use. The pve-enterprise repositoryis enabled by default:File /etc/apt/sources.list.d/pve-enterprise.listdeb https://enterprise.proxmox.com/debian/pve stretch pve-enterpriseAs soon as updates are available, the root@pam user is notified via email about the available new pack-ages. On the GUI, the change-log of each package can be viewed (if available), showing all details of theupdate. So you will never miss important security fixes.Please note that and you need a valid subscription key to access this repository. We offer different supportlevels, and you can find further details at http://www.proxmox.com/en/proxmox-ve/pricing.NoteYou can disable this repository by commenting out the above line using a # (at the start of the line).This prevents error messages if you do not have a subscription key. Please configure the pve-no-subscription repository in that case.3.1.2 Proxmox VE No-Subscription RepositoryAs the name suggests, you do not need a subscription key to access this repository. It can be used fortesting and non-production use. Its not recommended to run on production servers, as these packages arenot always heavily tested and validated.We recommend to configure this repository in /etc/apt/sources.list.File /etc/apt/sources.listdeb http://ftp.debian.org/debian stretch main contrib# PVE pve-no-subscription repository provided by proxmox.com,# NOT recommended for production usedeb http://download.proxmox.com/debian/pve stretch pve-no-subscription# security updatesdeb http://security.debian.org stretch/updates main contrib3.1.3 Proxmox VE Test RepositoryFinally, there is a repository called pvetest. This one contains the latest packages and is heavily used bydevelopers to test new features. As usual, you can configure this using /etc/apt/sources.list byadding the following line: Proxmox VE Administration Guide 307 / 328migration_unsecure: <boolean>Migration is secure using SSH tunnel by default. For secure private networks you can disable it tospeed up migration. Deprecated, use the migration property instead! Proxmox VE Administration Guide 308 / 328Appendix DFirewall Macro DefinitionsAmanda Amanda BackupAction proto dport sportPARAM udp 10080PARAM tcp 10080Auth Auth (identd) trafficAction proto dport sportPARAM tcp 113BGP Border Gateway Protocol trafficAction proto dport sportPARAM tcp 179BitTorrent BitTorrent traffic for BitTorrent 3.1 and earlierAction proto dport sportPARAM tcp 6881:6889PARAM udp 6881BitTorrent32 BitTorrent traffic for BitTorrent 3.2 and later Proxmox VE Administration Guide 309 / 328Action proto dport sportPARAM tcp 6881:6999PARAM udp 6881CVS Concurrent Versions System pserver trafficAction proto dport sportPARAM tcp 2401Ceph Ceph Storage Cluster traffic (Ceph Monitors, OSD & MDS Deamons)Action proto dport sportPARAM tcp 6789PARAM tcp 6800:7300Citrix Citrix/ICA traffic (ICA, ICA Browser, CGP)Action proto dport sportPARAM tcp 1494PARAM udp 1604PARAM tcp 2598DAAP Digital Audio Access Protocol traffic (iTunes, Rythmbox daemons)Action proto dport sportPARAM tcp 3689PARAM udp 3689DCC Distributed Checksum Clearinghouse spam filtering mechanismAction proto dport sportPARAM tcp 6277DHCPfwd Forwarded DHCP traffic Proxmox VE Administration Guide 310 / 328Action proto dport sportPARAM udp 67:68 67:68DHCPv6 DHCPv6 trafficAction proto dport sportPARAM udp 546:547 546:547DNS Domain Name System traffic (upd and tcp)Action proto dport sportPARAM udp 53PARAM tcp 53Distcc Distributed Compiler serviceAction proto dport sportPARAM tcp 3632FTP File Transfer ProtocolAction proto dport sportPARAM tcp 21Finger Finger protocol (RFC 742)Action proto dport sportPARAM tcp 79GNUnet GNUnet secure peer-to-peer networking trafficAction proto dport sportPARAM tcp 2086PARAM udp 2086PARAM tcp 1080 Proxmox VE Administration Guide 311 / 328Action proto dport sportPARAM udp 1080GRE Generic Routing Encapsulation tunneling protocolAction proto dport sportPARAM 47Git Git distributed revision control trafficAction proto dport sportPARAM tcp 9418HKP OpenPGP HTTP keyserver protocol trafficAction proto dport sportPARAM tcp 11371HTTP Hypertext Transfer Protocol (WWW)Action proto dport sportPARAM tcp 80HTTPS Hypertext Transfer Protocol (WWW) over SSLAction proto dport sportPARAM tcp 443ICPV2 Internet Cache Protocol V2 (Squid) trafficAction proto dport sportPARAM udp 3130ICQ AOL Instant Messenger traffic Proxmox VE Administration Guide 312 / 328Action proto dport sportPARAM tcp 5190IMAP Internet Message Access ProtocolAction proto dport sportPARAM tcp 143IMAPS Internet Message Access Protocol over SSLAction proto dport sportPARAM tcp 993IPIP IPIP capsulation trafficAction proto dport sportPARAM 94IPsec IPsec trafficAction proto dport sportPARAM udp 500 500PARAM 50IPsecah IPsec authentication (AH) trafficAction proto dport sportPARAM udp 500 500PARAM 51IPsecnat IPsec traffic and Nat-TraversalAction proto dport sportPARAM udp 500PARAM udp 4500 Proxmox VE Administration Guide 313 / 328Action proto dport sportPARAM 50IRC Internet Relay Chat trafficAction proto dport sportPARAM tcp 6667Jetdirect HP Jetdirect printingAction proto dport sportPARAM tcp 9100L2TP Layer 2 Tunneling Protocol trafficAction proto dport sportPARAM udp 1701LDAP Lightweight Directory Access Protocol trafficAction proto dport sportPARAM tcp 389LDAPS Secure Lightweight Directory Access Protocol trafficAction proto dport sportPARAM tcp 636MDNS Multicast DNSAction proto dport sportPARAM udp 5353MSNP Microsoft Notification Protocol Proxmox VE Administration Guide 314 / 328Action proto dport sportPARAM tcp 1863MSSQL Microsoft SQL ServerAction proto dport sportPARAM tcp 1433Mail Mail traffic (SMTP, SMTPS, Submission)Action proto dport sportPARAM tcp 25PARAM tcp 465PARAM tcp 587Munin Munin networked resource monitoring trafficAction proto dport sportPARAM tcp 4949MySQL MySQL serverAction proto dport sportPARAM tcp 3306NNTP NNTP traffic (Usenet).Action proto dport sportPARAM tcp 119NNTPS Encrypted NNTP traffic (Usenet)Action proto dport sportPARAM tcp 563 Proxmox VE Administration Guide 315 / 328NTP Network Time Protocol (ntpd)Action proto dport sportPARAM udp 123NeighborDiscoveryIPv6 neighbor solicitation, neighbor and router advertisementAction proto dport sportPARAM icmpv6 router-solicitationPARAM icmpv6 router-advertisementPARAM icmpv6 neighbor-solicitationPARAM icmpv6 neighbor-advertisementOSPF OSPF multicast trafficAction proto dport sportPARAM 89OpenVPN OpenVPN trafficAction proto dport sportPARAM udp 1194PCA Symantec PCAnywere (tm)Action proto dport sportPARAM udp 5632PARAM tcp 5631POP3 POP3 trafficAction proto dport sportPARAM tcp 110 Proxmox VE Administration Guide 316 / 328POP3S Encrypted POP3 trafficAction proto dport sportPARAM tcp 995PPtP Point-to-Point Tunneling ProtocolAction proto dport sportPARAM 47PARAM tcp 1723Ping ICMP echo requestAction proto dport sportPARAM icmp echo-requestPostgreSQL PostgreSQL serverAction proto dport sportPARAM tcp 5432Printer Line Printer protocol printingAction proto dport sportPARAM tcp 515RDP Microsoft Remote Desktop Protocol trafficAction proto dport sportPARAM tcp 3389RIP Routing Information Protocol (bidirectional) Proxmox VE Administration Guide 20 / 328sources.list entry for pvetestdeb http://download.proxmox.com/debian/pve stretch pvetestWarningthe pvetest repository should (as the name implies) only be used for testing new features or bugfixes.3.1.4 SecureAptWe use GnuPG to sign the Release files inside those repositories, and APT uses that signatures to verifythat all packages are from a trusted source.The key used for verification is already installed if you install from our installation CD. If you install by othermeans, you can manually download the key with:# wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg ←--O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpgPlease verify the checksum afterwards:# sha512sum /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpgffb95f0f4be68d2e753c8875ea2f8465864a58431d5361e88789568673551501ae574283a4e0492f17d79dc67edfb173a56a6304dea39e01f249ebdabc9f074a ←-/etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpgor# md5sum /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg511d36d0f1350c01c42a3dc9f3c27939 /etc/apt/trusted.gpg.d/proxmox-ve-release ←--5.x.gpg3.2 System Software UpdatesWe provide regular package updates on all repositories. You can install those update using the GUI, or youcan directly run the CLI command apt-get:apt-get updateapt-get dist-upgradeNoteThe apt package management system is extremely flexible and provides countless of feature - see manapt-get or [Hertzog13] for additional information.You should do such updates at regular intervals, or when we release versions with security related fixes.Major system upgrades are announced at the Proxmox VE Community Forum. Those announcement alsocontain detailed upgrade instructions.TipWe recommend to run regular upgrades, because it is important to get the latest security updates. Proxmox VE Administration Guide 317 / 328Action proto dport sportPARAM udp 520RNDC BIND remote management protocolAction proto dport sportPARAM tcp 953Razor Razor Antispam SystemAction proto dport sportACCEPT tcp 2703Rdate Remote time retrieval (rdate)Action proto dport sportPARAM tcp 37Rsync Rsync serverAction proto dport sportPARAM tcp 873SANE SANE network scanningAction proto dport sportPARAM tcp 6566SMB Microsoft SMB trafficAction proto dport sportPARAM udp 135,445PARAM udp 137:139PARAM udp 1024:65535 137PARAM tcp 135,139,445 Proxmox VE Administration Guide 318 / 328SMBswat Samba Web Administration ToolAction proto dport sportPARAM tcp 901SMTP Simple Mail Transfer ProtocolAction proto dport sportPARAM tcp 25SMTPS Encrypted Simple Mail Transfer ProtocolAction proto dport sportPARAM tcp 465SNMP Simple Network Management ProtocolAction proto dport sportPARAM udp 161:162PARAM tcp 161SPAMD Spam Assassin SPAMD trafficAction proto dport sportPARAM tcp 783SSH Secure shell trafficAction proto dport sportPARAM tcp 22SVN Subversion server (svnserve) Proxmox VE Administration Guide 319 / 328Action proto dport sportPARAM tcp 3690SixXS SixXS IPv6 Deployment and Tunnel BrokerAction proto dport sportPARAM tcp 3874PARAM udp 3740PARAM 41PARAM udp 5072,8374Squid Squid web proxy trafficAction proto dport sportPARAM tcp 3128Submission Mail message submission trafficAction proto dport sportPARAM tcp 587Syslog Syslog protocol (RFC 5424) trafficAction proto dport sportPARAM udp 514PARAM tcp 514TFTP Trivial File Transfer Protocol trafficAction proto dport sportPARAM udp 69Telnet Telnet traffic Proxmox VE Administration Guide 320 / 328Action proto dport sportPARAM tcp 23Telnets Telnet over SSLAction proto dport sportPARAM tcp 992Time RFC 868 Time protocolAction proto dport sportPARAM tcp 37Trcrt Traceroute (for up to 30 hops) trafficAction proto dport sportPARAM udp 33434:33524PARAM icmp echo-requestVNC VNC traffic for VNC display’s 0 - 99Action proto dport sportPARAM tcp 5900:5999VNCL VNC traffic from Vncservers to Vncviewers in listen modeAction proto dport sportPARAM tcp 5500Web WWW traffic (HTTP and HTTPS)Action proto dport sportPARAM tcp 80PARAM tcp 443 Proxmox VE Administration Guide 321 / 328Webcache Web Cache/Proxy traffic (port 8080)Action proto dport sportPARAM tcp 8080Webmin Webmin trafficAction proto dport sportPARAM tcp 10000Whois Whois (nicname, RFC 3912) trafficAction proto dport sportPARAM tcp 43 Proxmox VE Administration Guide 322 / 328Appendix EGNU Free Documentation LicenseVersion 1.3, 3 November 2008Copyright © 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc. http://fsf.org/Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is notallowed.0. PREAMBLEThe purpose of this License is to make a manual, textbook, or other functional and useful document "free"in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or withoutmodifying it, either commercially or noncommercially. Secondarily, this License preserves for the author andpublisher a way to get credit for their work, while not being considered responsible for modifications made byothers.This License is a kind of "copyleft", which means that derivative works of the document must themselvesbe free in the same sense. It complements the GNU General Public License, which is a copyleft licensedesigned for free software.We have designed this License in order to use it for manuals for free software, because free software needsfree documentation: a free program should come with manuals providing the same freedoms that the soft-ware does. But this License is not limited to software manuals; it can be used for any textual work, regardlessof subject matter or whether it is published as a printed book. We recommend this License principally forworks whose purpose is instruction or reference.1. APPLICABILITY AND DEFINITIONSThis License applies to any manual or other work, in any medium, that contains a notice placed by thecopyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The"Document", below, refers to any such manual or work. Any member of the public is a licensee, and isaddressed as "you". You accept the license if you copy, modify or distribute the work in a way requiringpermission under copyright law.A "Modified Version" of the Document means any work containing the Document or a portion of it, eithercopied verbatim, or with modifications and/or translated into another language.A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusivelywith the relationship of the publishers or authors of the Document to the Document’s overall subject (or Proxmox VE Administration Guide 323 / 328to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if theDocument is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.)The relationship could be a matter of historical connection with the subject or with related matters, or oflegal, commercial, philosophical, ethical or political position regarding them.The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those ofInvariant Sections, in the notice that says that the Document is released under this License. If a section doesnot fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Documentmay contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there arenone.The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-CoverTexts, in the notice that says that the Document is released under this License. A Front-Cover Text may beat most 5 words, and a Back-Cover Text may be at most 25 words.A "Transparent" copy of the Document means a machine-readable copy, represented in a format whosespecification is available to the general public, that is suitable for revising the document straightforwardlywith generic text editors or (for images composed of pixels) generic paint programs or (for drawings) somewidely available drawing editor, and that is suitable for input to text formatters or for automatic translation toa variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file formatwhose markup, or absence of markup, has been arranged to thwart or discourage subsequent modificationby readers is not Transparent. An image format is not Transparent if used for any substantial amount of text.A copy that is not "Transparent" is called "Opaque".Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format,LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML,PostScript or PDF designed for human modification. Examples of transparent image formats include PNG,XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietaryword processors, SGML or XML for which the DTD and/or processing tools are not generally available, andthe machine-generated HTML, PostScript or PDF produced by some word processors for output purposesonly.The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed tohold, legibly, the material this License requires to appear in the title page. For works in formats which do nothave any title page as such, "Title Page" means the text near the most prominent appearance of the work’stitle, preceding the beginning of the body of the text.The "publisher" means any person or entity that distributes copies of the Document to the public.A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ orcontains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands fora specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or"History".) To "Preserve the Title" of such a section when you modify the Document means that it remains asection "Entitled XYZ" according to this definition.The Document may include Warranty Disclaimers next to the notice which states that this License applies tothe Document. These Warranty Disclaimers are considered to be included by reference in this License, butonly as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have isvoid and has no effect on the meaning of this License.2. VERBATIM COPYINGYou may copy and distribute the Document in any medium, either commercially or noncommercially, providedthat this License, the copyright notices, and the license notice saying this License applies to the Document Proxmox VE Administration Guide 324 / 328are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. Youmay not use technical measures to obstruct or control the reading or further copying of the copies you makeor distribute. However, you may accept compensation in exchange for copies. If you distribute a large enoughnumber of copies you must also follow the conditions in section 3.You may also lend copies, under the same conditions stated above, and you may publicly display copies.3. COPYING IN QUANTITYIf you publish printed copies (or copies in media that commonly have printed covers) of the Document,numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose thecopies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, andBack-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisherof these copies. The front cover must present the full title with all words of the title equally prominent andvisible. You may add other material on the covers in addition. Copying with changes limited to the covers,as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatimcopying in other respects.If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (asmany as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.If you publish or distribute Opaque copies of the Document numbering more than 100, you must either includea machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copya computer-network location from which the general network-using public has access to download usingpublic-standard network protocols a complete Transparent copy of the Document, free of added material. Ifyou use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaquecopies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated locationuntil at least one year after the last time you distribute an Opaque copy (directly or through your agents orretailers) of that edition to the public.It is requested, but not required, that you contact the authors of the Document well before redistributing anylarge number of copies, to give them a chance to provide you with an updated version of the Document.4. MODIFICATIONSYou may copy and distribute a Modified Version of the Document under the conditions of sections 2 and3 above, provided that you release the Modified Version under precisely this License, with the ModifiedVersion filling the role of the Document, thus licensing distribution and modification of the Modified Versionto whoever possesses a copy of it. In addition, you must do these things in the Modified Version:A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and fromthose of previous versions (which should, if there were any, be listed in the History section of theDocument). You may use the same title as a previous version if the original publisher of that versiongives permission.B. List on the Title Page, as authors, one or more persons or entities responsible for authorship of themodifications in the Modified Version, together with at least five of the principal authors of the Docu-ment (all of its principal authors, if it has fewer than five), unless they release you from this requirement.C. State on the Title page the name of the publisher of the Modified Version, as the publisher.D. Preserve all the copyright notices of the Document. Proxmox VE Administration Guide 325 / 328E. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.F. Include, immediately after the copyright notices, a license notice giving the public permission to usethe Modified Version under the terms of this License, in the form shown in the Addendum below.G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in theDocument’s license notice.H. Include an unaltered copy of this License.I. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title,year, new authors, and publisher of the Modified Version as given on the Title Page. If there is nosection Entitled "History" in the Document, create one stating the title, year, authors, and publisher ofthe Document as given on its Title Page, then add an item describing the Modified Version as statedin the previous sentence.J. Preserve the network location, if any, given in the Document for public access to a Transparent copyof the Document, and likewise the network locations given in the Document for previous versions itwas based on. These may be placed in the "History" section. You may omit a network location for awork that was published at least four years before the Document itself, or if the original publisher ofthe version it refers to gives permission.K. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, andpreserve in the section all the substance and tone of each of the contributor acknowledgements and/ordedications given therein.L. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Sectionnumbers or the equivalent are not considered part of the section titles.M. Delete any section Entitled "Endorsements". Such a section may not be included in the ModifiedVersion.N. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any InvariantSection.O. Preserve any Warranty Disclaimers.If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sectionsand contain no material copied from the Document, you may at your option designate some or all of thesesections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’slicense notice. These titles must be distinct from any other section titles.You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Mod-ified Version by various parties—for example, statements of peer review or that the text has been approvedby an organization as the authoritative definition of a standard.You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as aBack-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity.If the Document already includes a cover text for the same cover, previously added by you or by arrangementmade by the same entity you are acting on behalf of, you may not add another; but you may replace the oldone, on explicit permission from the previous publisher that added the old one.The author(s) and publisher(s) of the Document do not by this License give permission to use their namesfor publicity for or to assert or imply endorsement of any Modified Version. Proxmox VE Administration Guide 326 / 3285. COMBINING DOCUMENTSYou may combine the Document with other documents released under this License, under the terms definedin section 4 above for modified versions, provided that you include in the combination all of the InvariantSections of all of the original documents, unmodified, and list them all as Invariant Sections of your combinedwork in its license notice, and that you preserve all their Warranty Disclaimers.The combined work need only contain one copy of this License, and multiple identical Invariant Sectionsmay be replaced with a single copy. If there are multiple Invariant Sections with the same name but differentcontents, make the title of each such section unique by adding at the end of it, in parentheses, the name ofthe original author or publisher of that section if known, or else a unique number. Make the same adjustmentto the section titles in the list of Invariant Sections in the license notice of the combined work.In the combination, you must combine any sections Entitled "History" in the various original documents,forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and anysections Entitled "Dedications". You must delete all sections Entitled "Endorsements".6. COLLECTIONS OF DOCUMENTSYou may make a collection consisting of the Document and other documents released under this License,and replace the individual copies of this License in the various documents with a single copy that is in-cluded in the collection, provided that you follow the rules of this License for verbatim copying of each of thedocuments in all other respects.You may extract a single document from such a collection, and distribute it individually under this License,provided you insert a copy of this License into the extracted document, and follow this License in all otherrespects regarding verbatim copying of that document.7. AGGREGATION WITH INDEPENDENT WORKSA compilation of the Document or its derivatives with other separate and independent documents or works,in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting fromthe compilation is not used to limit the legal rights of the compilation’s users beyond what the individual workspermit. When the Document is included in an aggregate, this License does not apply to the other works inthe aggregate which are not themselves derivative works of the Document.If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Documentis less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers thatbracket the Document within the aggregate, or the electronic equivalent of covers if the Document is inelectronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.8. TRANSLATIONTranslation is considered a kind of modification, so you may distribute translations of the Document underthe terms of section 4. Replacing Invariant Sections with translations requires special permission from theircopyright holders, but you may include translations of some or all Invariant Sections in addition to the originalversions of these Invariant Sections. You may include a translation of this License, and all the license noticesin the Document, and any Warranty Disclaimers, provided that you also include the original English version ofthis License and the original versions of those notices and disclaimers. In case of a disagreement betweenthe translation and the original version of this License or a notice or disclaimer, the original version willprevail. Proxmox VE Administration Guide 21 / 3283.3 Network ConfigurationNetwork configuration can be done either via the GUI, or by manually editing the file /etc/network/interfaces, which contains the whole network configuration. The interfaces(5) manual pagecontains the complete format description. All Proxmox VE tools try hard to keep direct user modifications,but using the GUI is still preferable, because it protects you from errors.Once the network is configured, you can use the Debian traditional tools ifup and ifdown commands tobring interfaces up and down.NoteProxmox VE does not write changes directly to /etc/network/interfaces. Instead, we write intoa temporary file called /etc/network/interfaces.new, and commit those changes when youreboot the node.3.3.1 Naming ConventionsWe currently use the following naming conventions for device names:• Ethernet devices: en*, systemd network interface names. This naming scheme is used for new ProxmoxVE installations since version 5.0.• Ethernet devices: eth[N], where 0 ≤ N (eth0, eth1, . . . ) This naming scheme is used for Proxmox VEhosts which were installed before the 5.0 release. When upgrading to 5.0, the names are kept as-is.• Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (vmbr0 - vmbr4094)• Bonds: bond[N], where 0 ≤ N (bond0, bond1, . . . )• VLANs: Simply add the VLAN number to the device name, separated by a period (eno1.50, bond1.30)This makes it easier to debug networks problems, because the device name implies the device type.Systemd Network Interface NamesSystemd uses the two character prefix en for Ethernet network devices. The next characters depends on thedevice driver and the fact which schema matches first.• o<index>[n<phys_port_name>|d<dev_port>] — devices on board• s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id• [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id• x<MAC> — device by MAC addressThe most common patterns are:• eno1 — is the first on board NIC• enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1.For more information see Predictable Network Interface Names. Proxmox VE Administration Guide 327 / 328If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement(section 4) to Preserve its Title (section 1) will typically require changing the actual title.9. TERMINATIONYou may not copy, modify, sublicense, or distribute the Document except as expressly provided under thisLicense. Any attempt otherwise to copy, modify, sublicense, or distribute it is void, and will automaticallyterminate your rights under this License.However, if you cease all violation of this License, then your license from a particular copyright holder isreinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license,and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable meansprior to 60 days after the cessation.Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holdernotifies you of the violation by some reasonable means, this is the first time you have received notice ofviolation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 daysafter your receipt of the notice.Termination of your rights under this section does not terminate the licenses of parties who have receivedcopies or rights from you under this License. If your rights have been terminated and not permanentlyreinstated, receipt of a copy of some or all of the same material does not give you any rights to use it.10. FUTURE REVISIONS OF THIS LICENSEThe Free Software Foundation may publish new, revised versions of the GNU Free Documentation Licensefrom time to time. Such new versions will be similar in spirit to the present version, but may differ in detail toaddress new problems or concerns. See http://www.gnu.org/copyleft/.Each version of the License is given a distinguishing version number. If the Document specifies that aparticular numbered version of this License "or any later version" applies to it, you have the option of followingthe terms and conditions either of that specified version or of any later version that has been published (notas a draft) by the Free Software Foundation. If the Document does not specify a version number of thisLicense, you may choose any version ever published (not as a draft) by the Free Software Foundation. Ifthe Document specifies that a proxy can decide which future versions of this License can be used, thatproxy’s public statement of acceptance of a version permanently authorizes you to choose that version forthe Document.11. RELICENSING"Massive Multiauthor Collaboration Site" (or "MMC Site") means any World Wide Web server that publishescopyrightable works and also provides prominent facilities for anybody to edit those works. A public wikithat anybody can edit is an example of such a server. A "Massive Multiauthor Collaboration" (or "MMC")contained in the site means any set of copyrightable works thus published on the MMC site."CC-BY-SA" means the Creative Commons Attribution-Share Alike 3.0 license published by Creative Com-mons Corporation, a not-for-profit corporation with a principal place of business in San Francisco, California,as well as future copyleft versions of that license published by that same organization."Incorporate" means to publish or republish a Document, in whole or in part, as part of another Document.An MMC is "eligible for relicensing" if it is licensed under this License, and if all works that were first publishedunder this License somewhere other than this MMC, and subsequently incorporated in whole or in part into Proxmox VE Administration Guide 328 / 328the MMC, (1) had no cover texts or invariant sections, and (2) were thus incorporated prior to November 1,2008.The operator of an MMC Site may republish an MMC contained in the site under CC-BY-SA on the same siteat any time before August 1, 2009, provided the MMC is eligible for relicensing. Proxmox VE Administration Guide 22 / 3283.3.2 Choosing a network configurationDepending on your current network organization and your resources you can choose either a bridged, routed,or masquerading networking setup.Proxmox VE server in a private LAN, using an external gateway to reach the internetThe Bridged model makes the most sense in this case, and this is also the default mode on new ProxmoxVE installations. Each of your Guest system will have a virtual interface attached to the Proxmox VE bridge.This is similar in effect to having the Guest network card directly connected to a new switch on your LAN, theProxmox VE host playing the role of the switch.Proxmox VE server at hosting provider, with public IP ranges for GuestsFor this setup, you can use either a Bridged or Routed model, depending on what your provider allows.Proxmox VE server at hosting provider, with a single public IP addressIn that case the only way to get outgoing network accesses for your guest systems is to use Masquerading.For incoming network access to your guests, you will need to configure Port Forwarding.For further flexibility, you can configure VLANs (IEEE 802.1q) and network bonding, also known as "linkaggregation". That way it is possible to build complex and flexible virtual networks.3.3.3 Default Configuration using a BridgeBridges are like physical network switches implemented in software. All VMs can share a single bridge, oryou can create multiple bridges to separate network domains. Each host can have up to 4094 bridges.The installation program creates a single bridge named vmbr0, which is connected to the first Ethernetcard. The corresponding configuration in /etc/network/interfaces might look like this:auto loiface lo inet loopbackiface eno1 inet manualauto vmbr0iface vmbr0 inet staticaddress 192.168.10.2netmask 255.255.255.0gateway 192.168.10.1bridge_ports eno1bridge_stp offbridge_fd 0Virtual machines behave as if they were directly connected to the physical network. The network, in turn,sees each virtual machine as having its own MAC, even though there is only one network cable connectingall of these VMs to the network. Proxmox VE Administration Guide 23 / 3283.3.4 Routed ConfigurationMost hosting providers do not support the above setup. For security reasons, they disable networking assoon as they detect multiple MAC addresses on a single interface.TipSome providers allows you to register additional MACs on there management interface. This avoids theproblem, but is clumsy to configure because you need to register a MAC for each of your VMs.You can avoid the problem by “routing” all traffic via a single interface. This makes sure that all networkpackets use the same MAC address.A common scenario is that you have a public IP (assume 198.51.100.5 for this example), and an addi-tional IP block for your VMs (203.0.113.16/29). We recommend the following setup for such situations:auto loiface lo inet loopbackauto eno1iface eno1 inet staticaddress 198.51.100.5netmask 255.255.255.0gateway 198.51.100.1post-up echo 1 > /proc/sys/net/ipv4/ip_forwardpost-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arpauto vmbr0iface vmbr0 inet staticaddress 203.0.113.17netmask 255.255.255.248bridge_ports nonebridge_stp offbridge_fd 03.3.5 Masquerading (NAT) with iptablesMasquerading allows guests having only a private IP address to access the network by using the host IPaddress for outgoing traffic. Each outgoing packet is rewritten by iptables to appear as originating fromthe host, and responses are rewritten accordingly to be routed to the original sender.auto loiface lo inet loopbackauto eno1#real IP addressiface eno1 inet staticaddress 198.51.100.5netmask 255.255.255.0gateway 198.51.100.1 Proxmox VE Administration Guide 24 / 328auto vmbr0#private sub networkiface vmbr0 inet staticaddress 10.10.10.1netmask 255.255.255.0bridge_ports nonebridge_stp offbridge_fd 0post-up echo 1 > /proc/sys/net/ipv4/ip_forwardpost-up iptables -t nat -A POSTROUTING -s ’10.10.10.0/24’ -o eno1 ←--j MASQUERADEpost-down iptables -t nat -D POSTROUTING -s ’10.10.10.0/24’ -o eno1 ←--j MASQUERADE3.3.6 Linux BondBonding (also called NIC teaming or Link Aggregation) is a technique for binding multiple NIC’s to a singlenetwork device. It is possible to achieve different goals, like make the network fault-tolerant, increase theperformance or both together.High-speed hardware like Fibre Channel and the associated switching hardware can be quite expensive. Bydoing link aggregation, two NICs can appear as one logical interface, resulting in double speed. This is anative Linux kernel feature that is supported by most switches. If your nodes have multiple Ethernet ports,you can distribute your points of failure by running network cables to different switches and the bondedconnection will failover to one cable or the other in case of network trouble.Aggregated links can improve live-migration delays and improve the speed of replication of data betweenProxmox VE Cluster nodes.There are 7 modes for bonding:• Round-robin (balance-rr): Transmit network packets in sequential order from the first available networkinterface (NIC) slave through the last. This mode provides load balancing and fault tolerance.• Active-backup (active-backup): Only one NIC slave in the bond is active. A different slave becomesactive if, and only if, the active slave fails. The single logical bonded interface’s MAC address is externallyvisible on only one NIC (port) to avoid distortion in the network switch. This mode provides fault tolerance.• XOR (balance-xor): Transmit network packets based on [(source MAC address XOR’d with destinationMAC address) modulo NIC slave count]. This selects the same NIC slave for each destination MACaddress. This mode provides load balancing and fault tolerance.• Broadcast (broadcast): Transmit network packets on all slave network interfaces. This mode providesfault tolerance.• IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP): Creates aggregation groups that share thesame speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group ac-cording to the 802.3ad specification.• Adaptive transmit load balancing (balance-tlb): Linux bonding driver mode that does not require anyspecial network-switch support. The outgoing network packet traffic is distributed according to the current Proxmox VE Administration Guide 25 / 328load (computed relative to the speed) on each network interface slave. Incoming traffic is received by onecurrently designated slave network interface. If this receiving slave fails, another slave takes over the MACaddress of the failed receiving slave.• Adaptive load balancing (balance-alb): Includes balance-tlb plus receive load balancing (rlb) for IPV4traffic, and does not require any special network switch support. The receive load balancing is achieved byARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way outand overwrites the source hardware address with the unique hardware address of one of the NIC slaves inthe single logical bonded interface such that different network-peers use different MAC addresses for theirnetwork packet traffic.If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using the correspondingbonding mode (802.3ad). Otherwise you should generally use the active-backup mode.If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive modeon the bonding interfaces, other modes are unsupported.The following bond configuration can be used as distributed/shared storage network. The benefit would bethat you get more speed and the network will be fault-tolerant.Example: Use bond with fixed IP addressauto loiface lo inet loopbackiface eno1 inet manualiface eno2 inet manualauto bond0iface bond0 inet staticslaves eno1 eno2address 192.168.1.2netmask 255.255.255.0bond_miimon 100bond_mode 802.3adbond_xmit_hash_policy layer2+3auto vmbr0iface vmbr0 inet staticaddress 10.10.10.2netmask 255.255.255.0gateway 10.10.10.1bridge_ports eno1bridge_stp offbridge_fd 0Another possibility it to use the bond directly as bridge port. This can be used to make the guest networkfault-tolerant.Example: Use a bond as bridge port Proxmox VE Administration Guide 26 / 328auto loiface lo inet loopbackiface eno1 inet manualiface eno2 inet manualauto bond0iface bond0 inet manualslaves eno1 eno2bond_miimon 100bond_mode 802.3adbond_xmit_hash_policy layer2+3auto vmbr0iface vmbr0 inet staticaddress 10.10.10.2netmask 255.255.255.0gateway 10.10.10.1bridge_ports bond0bridge_stp offbridge_fd 03.3.7 VLAN 802.1QA virtual LAN (VLAN) is a broadcast domain that is partitioned and isolated in the network at layer two. So itis possible to have multiple networks (4096) in a physical network, each independent of the other ones.Each VLAN network is identified by a number often called tag. Network packages are then tagged to identifywhich virtual network they belong to.VLAN for Guest NetworksProxmox VE supports this setup out of the box. You can specify the VLAN tag when you create a VM.The VLAN tag is part of the guest network confinuration. The networking layer supports differnet modes toimplement VLANs, depending on the bridge configuration:• VLAN awareness on the Linux bridge: In this case, each guest’s virtual network card is assigned toa VLAN tag, which is transparently supported by the Linux bridge. Trunk mode is also possible, but thatmakes the configuration in the guest necessary.• "traditional" VLAN on the Linux bridge: In contrast to the VLAN awareness method, this method is nottransparent and creates a VLAN device with associated bridge for each VLAN. That is, if e.g. in our defaultnetwork, a guest VLAN 5 is used to create eno1.5 and vmbr0v5, which remains until rebooting.• Open vSwitch VLAN: This mode uses the OVS VLAN feature.• Guest configured VLAN: VLANs are assigned inside the guest. In this case, the setup is completelydone inside the guest and can not be influenced from the outside. The benefit is that you can use morethan one VLAN on a single virtual NIC. Proxmox VE Administration Guide iv2.4 Install from USB Stick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.4.1 Prepare a USB flash drive as install medium . . . . . . . . . . . . . . . . . . . . . . . 162.4.2 Instructions for GNU/Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.4.3 Instructions for OSX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.4.4 Instructions for Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.4.5 Boot your server from USB media . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Host System Administration 183.1 Package Repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.1.1 Proxmox VE Enterprise Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.2 Proxmox VE No-Subscription Repository . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.3 Proxmox VE Test Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.4 SecureApt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2 System Software Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.3 Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.3.1 Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.3.2 Choosing a network configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3.3 Default Configuration using a Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3.4 Routed Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.3.5 Masquerading (NAT) with iptables . . . . . . . . . . . . . . . . . . . . . . . . . . 233.3.6 Linux Bond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.3.7 VLAN 802.1Q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.4 Time Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.4.1 Using Custom NTP Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.5 External Metric Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.5.1 Graphite server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.5.2 Influxdb plugin configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.6 Disk Health Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.7 Logical Volume Manager (LVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.7.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.7.2 Bootloader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.7.3 Creating a Volume Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.7.4 Creating an extra LV for /var/lib/vz . . . . . . . . . . . . . . . . . . . . . . . . 323.7.5 Resizing the thin pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.7.6 Create a LVM-thin pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.8 ZFS on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Proxmox VE Administration Guide 27 / 328VLAN on the HostTo allow host communication with an isolated network. It is possible to apply VLAN tags to any networkdevice (NIC, Bond, Bridge). In general, you should configure the VLAN on the interface with the leastabstraction layers between itself and the physical NIC.For example, in a default configuration where you want to place the host management address on a separateVLAN.NoteIn the examples we use the VLAN at bridge level to ensure the correct function of VLAN 5 in the guestnetwork, but in combination with VLAN anwareness bridge this it will not work for guest network VLAN 5.The downside of this setup is more CPU usage.Example: Use VLAN 5 for the Proxmox VE management IPauto loiface lo inet loopbackiface eno1 inet manualiface eno1.5 inet manualauto vmbr0v5iface vmbr0v5 inet staticaddress 10.10.10.2netmask 255.255.255.0gateway 10.10.10.1bridge_ports eno1.5bridge_stp offbridge_fd 0auto vmbr0iface vmbr0 inet manualbridge_ports eno1bridge_stp offbridge_fd 0The next example is the same setup but a bond is used to make this network fail-safe.Example: Use VLAN 5 with bond0 for the Proxmox VE management IPauto loiface lo inet loopbackiface eno1 inet manualiface eno2 inet manualauto bond0 Proxmox VE Administration Guide 28 / 328iface bond0 inet manualslaves eno1 eno2bond_miimon 100bond_mode 802.3adbond_xmit_hash_policy layer2+3iface bond0.5 inet manualauto vmbr0v5iface vmbr0v5 inet staticaddress 10.10.10.2netmask 255.255.255.0gateway 10.10.10.1bridge_ports bond0.5bridge_stp offbridge_fd 0auto vmbr0iface vmbr0 inet manualbridge_ports bond0bridge_stp offbridge_fd 03.4 Time SynchronizationThe Proxmox VE cluster stack itself relies heavily on the fact that all the nodes have precisely synchronizedtime. Some other components, like Ceph, also refuse to work properly if the local time on nodes is not insync.Time synchronization between nodes can be achieved with the “Network Time Protocol” (NTP). Proxmox VEuses systemd-timesyncd as NTP client by default, preconfigured to use a set of public servers. Thissetup works out of the box in most cases.3.4.1 Using Custom NTP ServersIn some cases, it might be desired to not use the default NTP servers. For example, if your Proxmox VEnodes do not have access to the public internet (e.g., because of restrictive firewall rules), you need to setuplocal NTP servers and tell systemd-timesyncd to use them:File /etc/systemd/timesyncd.conf[Time]Servers=ntp1.example.com ntp2.example.com ntp3.example.com ntp4.example.comAfter restarting the synchronization service (systemctl restart systemd-timesyncd) you shouldverify that your newly configured NTP servers are used by checking the journal (journalctl --since-1h -u systemd-timesyncd): Proxmox VE Administration Guide 29 / 328...Oct 07 14:58:36 node1 systemd[1]: Stopping Network Time Synchronization...Oct 07 14:58:36 node1 systemd[1]: Starting Network Time Synchronization...Oct 07 14:58:36 node1 systemd[1]: Started Network Time Synchronization.Oct 07 14:58:36 node1 systemd-timesyncd[13514]: Using NTP server ←-10.0.0.1:123 (ntp1.example.com).Oct 07 14:58:36 nora systemd-timesyncd[13514]: interval/delta/delay/jitter/ ←-drift 64s/-0.002s/0.020s/0.000s/-31ppm...3.5 External Metric ServerStarting with Proxmox VE 4.0, you can define external metric servers, which will be sent various stats aboutyour hosts, virtual machines and storages.Currently supported are:• graphite (see http://graphiteapp.org )• influxdb (see https://www.influxdata.com/time-series-platform/influxdb/ )The server definitions are saved in /etc/pve/status.cfg3.5.1 Graphite server configurationThe definition of a server is:graphite:server your-serverport your-portpath your-pathwhere your-port defaults to 2003 and your-path defaults to proxmoxProxmox VE sends the data over udp, so the graphite server has to be configured for this3.5.2 Influxdb plugin configurationThe definition is:influxdb:server your-serverport your-portProxmox VE sends the data over udp, so the influxdb server has to be configured for thisHere is an example configuration for influxdb (on your influxdb server): Proxmox VE Administration Guide 30 / 328[[udp]]enabled = truebind-address = "0.0.0.0:8089"database = "proxmox"batch-size = 1000batch-timeout = "1s"With this configuration, your server listens on all IP addresses on port 8089, and writes the data in theproxmox database3.6 Disk Health MonitoringAlthough a robust and redundant storage is recommended, it can be very helpful to monitor the health ofyour local disks.Starting with Proxmox VE 4.3, the package smartmontools1is installed and required. This is a set of toolsto monitor and control the S.M.A.R.T. system for local hard disks.You can get the status of a disk by issuing the following command:# smartctl -a /dev/sdXwhere /dev/sdX is the path to one of your local disks.If the output says:SMART support is: Disabledyou can enable it with the command:# smartctl -s on /dev/sdXFor more information on how to use smartctl, please see man smartctl.By default, smartmontools daemon smartd is active and enabled, and scans the disks under /dev/sdX and/dev/hdX every 30 minutes for errors and warnings, and sends an e-mail to root if it detects a problem.For more information about how to configure smartd, please see man smartd and man smartd.conf.If you use your hard disks with a hardware raid controller, there are most likely tools to monitor the disks inthe raid array and the array itself. For more information about this, please refer to the vendor of your raidcontroller.3.7 Logical Volume Manager (LVM)Most people install Proxmox VE directly on a local disk. The Proxmox VE installation CD offers severaloptions for local disk management, and the current default setup uses LVM. The installer let you select asingle disk for such setup, and uses that disk as physical volume for the Volume Group (VG) pve. Thefollowing output is from a test installation using a small 8GB disk:1smartmontools homepage https://www.smartmontools.org Proxmox VE Administration Guide 31 / 328# pvsPV VG Fmt Attr PSize PFree/dev/sda3 pve lvm2 a-- 7.87g 876.00m# vgsVG #PV #LV #SN Attr VSize VFreepve 1 3 0 wz--n- 7.87g 876.00mThe installer allocates three Logical Volumes (LV) inside this VG:# lvsLV VG Attr LSize Pool Origin Data% Meta%data pve twi-a-tz-- 4.38g 0.00 0.63root pve -wi-ao---- 1.75gswap pve -wi-ao---- 896.00mrootFormatted as ext4, and contains the operation system.swapSwap partitiondataThis volume uses LVM-thin, and is used to store VM images. LVM-thin is preferable for this task,because it offers efficient support for snapshots and clones.For Proxmox VE versions up to 4.1, the installer creates a standard logical volume called “data”, which ismounted at /var/lib/vz.Starting from version 4.2, the logical volume “data” is a LVM-thin pool, used to store block based guestimages, and /var/lib/vz is simply a directory on the root file system.3.7.1 HardwareWe highly recommend to use a hardware RAID controller (with BBU) for such setups. This increases perfor-mance, provides redundancy, and make disk replacements easier (hot-pluggable).LVM itself does not need any special hardware, and memory requirements are very low.3.7.2 BootloaderWe install two boot loaders by default. The first partition contains the standard GRUB boot loader. Thesecond partition is an EFI System Partition (ESP), which makes it possible to boot on EFI systems.3.7.3 Creating a Volume GroupLet’s assume we have an empty disk /dev/sdb, onto which we want to create a volume group named“vmdata”. Proxmox VE Administration Guide 32 / 328CautionPlease note that the following commands will destroy all existing data on /dev/sdb.First create a partition.# sgdisk -N 1 /dev/sdbCreate a Physical Volume (PV) without confirmation and 250K metadatasize.# pvcreate --metadatasize 250k -y -ff /dev/sdb1Create a volume group named “vmdata” on /dev/sdb1# vgcreate vmdata /dev/sdb13.7.4 Creating an extra LV for /var/lib/vzThis can be easily done by creating a new thin LV.# lvcreate -n <Name> -V <Size[M,G,T]> <VG>/<LVThin_pool>A real world example:# lvcreate -n vz -V 10G pve/dataNow a filesystem must be created on the LV.# mkfs.ext4 /dev/pve/vzAt last this has to be mounted.Warningbe sure that /var/lib/vz is empty. On a default installation it’s not.To make it always accessible add the following line in /etc/fstab.# echo ’/dev/pve/vz /var/lib/vz ext4 defaults 0 2’ >> /etc/fstab3.7.5 Resizing the thin poolResize the LV and the metadata pool can be achieved with the following command.# lvresize --size +<size[\M,G,T]> --poolmetadatasize +<size[\M,G]> < ←-VG>/<LVThin_pool>NoteWhen extending the data pool, the metadata pool must also be extended. Proxmox VE Administration Guide 33 / 3283.7.6 Create a LVM-thin poolA thin pool has to be created on top of a volume group. How to create a volume group see Section LVM.# lvcreate -L 80G -T -n vmstore vmdata3.8 ZFS on LinuxZFS is a combined file system and logical volume manager designed by Sun Microsystems. Starting withProxmox VE 3.4, the native Linux kernel port of the ZFS file system is introduced as optional file system andalso as an additional selection for the root file system. There is no need for manually compile ZFS modules- all packages are included.By using ZFS, its possible to achieve maximum enterprise features with low budget hardware, but also highperformance systems by leveraging SSD caching or even SSD only setups. ZFS can replace cost intensehardware raid cards by moderate CPU and memory load combined with easy management.GENERAL ZFS ADVANTAGES• Easy configuration and management with Proxmox VE GUI and CLI.• Reliable• Protection against data corruption• Data compression on file system level• Snapshots• Copy-on-write clone• Various raid levels: RAID0, RAID1, RAID10, RAIDZ-1, RAIDZ-2 and RAIDZ-3• Can use SSD for cache• Self healing• Continuous integrity checking• Designed for high storage capacities• Protection against data corruption• Asynchronous replication over network• Open Source• Encryption• . . . Proxmox VE Administration Guide 34 / 3283.8.1 HardwareZFS depends heavily on memory, so you need at least 8GB to start. In practice, use as much you can getfor your hardware/budget. To prevent data corruption, we recommend the use of high quality ECC RAM.If you use a dedicated cache and/or log disk, you should use an enterprise class SSD (e.g. Intel SSD DCS3700 Series). This can increase the overall performance significantly.ImportantDo not use ZFS on top of hardware controller which has its own cache management. ZFS needs todirectly communicate with disks. An HBA adapter is the way to go, or something like LSI controllerflashed in “IT” mode.If you are experimenting with an installation of Proxmox VE inside a VM (Nested Virtualization), don’t usevirtio for disks of that VM, since they are not supported by ZFS. Use IDE or SCSI instead (works alsowith virtio SCSI controller type).3.8.2 Installation as Root File SystemWhen you install using the Proxmox VE installer, you can choose ZFS for the root file system. You need toselect the RAID type at installation time:RAID0 Also called “striping”. The capacity of such volume is the sum of the capacities of alldisks. But RAID0 does not add any redundancy, so the failure of a single drivemakes the volume unusable.RAID1 Also called “mirroring”. Data is written identically to all disks. This mode requires atleast 2 disks with the same size. The resulting capacity is that of a single disk.RAID10 A combination of RAID0 and RAID1. Requires at least 4 disks.RAIDZ-1 A variation on RAID-5, single parity. Requires at least 3 disks.RAIDZ-2 A variation on RAID-5, double parity. Requires at least 4 disks.RAIDZ-3 A variation on RAID-5, triple parity. Requires at least 5 disks.The installer automatically partitions the disks, creates a ZFS pool called rpool, and installs the root filesystem on the ZFS subvolume rpool/ROOT/pve-1.Another subvolume called rpool/data is created to store VM images. In order to use that with theProxmox VE tools, the installer creates the following configuration entry in /etc/pve/storage.cfg:zfspool: local-zfspool rpool/datasparsecontent images,rootdirAfter installation, you can view your ZFS pool status using the zpool command: Proxmox VE Administration Guide 35 / 328# zpool statuspool: rpoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMrpool ONLINE 0 0 0mirror-0 ONLINE 0 0 0sda2 ONLINE 0 0 0sdb2 ONLINE 0 0 0mirror-1 ONLINE 0 0 0sdc ONLINE 0 0 0sdd ONLINE 0 0 0errors: No known data errorsThe zfs command is used configure and manage your ZFS file systems. The following command lists allfile systems after installation:# zfs listNAME USED AVAIL REFER MOUNTPOINTrpool 4.94G 7.68T 96K /rpoolrpool/ROOT 702M 7.68T 96K /rpool/ROOTrpool/ROOT/pve-1 702M 7.68T 702M /rpool/data 96K 7.68T 96K /rpool/datarpool/swap 4.25G 7.69T 64K -3.8.3 BootloaderThe default ZFS disk partitioning scheme does not use the first 2048 sectors. This gives enough room toinstall a GRUB boot partition. The Proxmox VE installer automatically allocates that space, and installs theGRUB boot loader there. If you use a redundant RAID setup, it installs the boot loader on all disk requiredfor booting. So you can boot even if some disks fail.NoteIt is not possible to use ZFS as root file system with UEFI boot.3.8.4 ZFS AdministrationThis section gives you some usage examples for common tasks. ZFS itself is really powerful and providesmany options. The main commands to manage ZFS are zfs and zpool. Both commands come with greatmanual pages, which can be read with:# man zpool# man zfs Proxmox VE Administration Guide 36 / 328Create a new zpoolTo create a new pool, at least one disk is needed. The ashift should have the same sector-size (2 powerof ashift) or larger as the underlying disk.zpool create -f -o ashift=12 <pool> <device>To activate compressionzfs set compression=lz4 <pool>Create a new pool with RAID-0Minimum 1 Diskzpool create -f -o ashift=12 <pool> <device1> <device2>Create a new pool with RAID-1Minimum 2 Diskszpool create -f -o ashift=12 <pool> mirror <device1> <device2>Create a new pool with RAID-10Minimum 4 Diskszpool create -f -o ashift=12 <pool> mirror <device1> <device2> ←-mirror <device3> <device4>Create a new pool with RAIDZ-1Minimum 3 Diskszpool create -f -o ashift=12 <pool> raidz1 <device1> <device2> < ←-device3>Create a new pool with RAIDZ-2Minimum 4 Diskszpool create -f -o ashift=12 <pool> raidz2 <device1> <device2> < ←-device3> <device4>Create a new pool with cache (L2ARC)It is possible to use a dedicated cache drive partition to increase the performance (use SSD).As <device> it is possible to use more devices, like it’s shown in "Create a new pool with RAID*".zpool create -f -o ashift=12 <pool> <device> cache <cache_device> Proxmox VE Administration Guide v3.8.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.8.2 Installation as Root File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.8.3 Bootloader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.8.4 ZFS Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.8.5 Activate E-Mail Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.8.6 Limit ZFS Memory Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.9 Certificate Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.9.1 Certificates for communication within the cluster . . . . . . . . . . . . . . . . . . . . . 393.9.2 Certificates for API and web GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 Hyper-converged Infrastructure 424.1 Benefits of a Hyper-Converged Infrastructure (HCI) with Proxmox VE . . . . . . . . . . . . . . 424.2 Manage Ceph Services on Proxmox VE Nodes . . . . . . . . . . . . . . . . . . . . . . . . . 434.2.1 Precondition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.2.2 Installation of Ceph Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.2.3 Creating initial Ceph configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.2.4 Creating Ceph Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.2.5 Creating Ceph Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.2.6 Creating Ceph OSDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.2.7 Creating Ceph Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.2.8 Ceph CRUSH & device classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.2.9 Ceph Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525 Graphical User Interface 535.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.2 Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.3 GUI Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.3.1 Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.3.2 Resource Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.3.3 Log Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.4 Content Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.4.1 Datacenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.4.2 Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.4.3 Guests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.4.4 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615.4.5 Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Proxmox VE Administration Guide 37 / 328Create a new pool with log (ZIL)It is possible to use a dedicated cache drive partition to increase the performance(SSD).As <device> it is possible to use more devices, like it’s shown in "Create a new pool with RAID*".zpool create -f -o ashift=12 <pool> <device> log <log_device>Add cache and log to an existing poolIf you have an pool without cache and log. First partition the SSD in 2 partition with parted or gdiskImportantAlways use GPT partition tables.The maximum size of a log device should be about half the size of physical memory, so this is usually quitesmall. The rest of the SSD can be used as cache.zpool add -f <pool> log <device-part1> cache <device-part2>Changing a failed devicezpool replace -f <pool> <old device> <new-device>3.8.5 Activate E-Mail NotificationZFS comes with an event daemon, which monitors events generated by the ZFS kernel module. The daemoncan also send emails on ZFS events like pool errors. Newer ZFS packages ships the daemon in a separatepackage, and you can install it using apt-get:# apt-get install zfs-zedTo activate the daemon it is necessary to edit /etc/zfs/zed.d/zed.rc with your favourite editor, anduncomment the ZED_EMAIL_ADDR setting:ZED_EMAIL_ADDR="root"Please note Proxmox VE forwards mails to root to the email address configured for the root user.ImportantThe only setting that is required is ZED_EMAIL_ADDR. All other settings are optional. Proxmox VE Administration Guide 38 / 3283.8.6 Limit ZFS Memory UsageIt is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to preventperformance shortage of the host. Use your preferred editor to change the configuration in /etc/modprobe.d/zfs.conf and insert:options zfs zfs_arc_max=8589934592This example setting limits the usage to 8GB.ImportantIf your root file system is ZFS you must update your initramfs every time this value changes:update-initramfs -uSWAP on ZFSSWAP on ZFS on Linux may generate some troubles, like blocking the server or generating a high IO load,often seen when starting a Backup to an external Storage.We strongly recommend to use enough memory, so that you normally do not run into low memory situations.Additionally, you can lower the “swappiness” value. A good value for servers is 10:sysctl -w vm.swappiness=10To make the swappiness persistent, open /etc/sysctl.conf with an editor of your choice and add thefollowing line:vm.swappiness = 10Table 3.1: Linux kernel swappiness parameter valuesValue Strategyvm.swappiness = 0 The kernel will swap only to avoid an out of memory conditionvm.swappiness = 1 Minimum amount of swapping without disabling it entirely.vm.swappiness = 10 This value is sometimes recommended to improve performancewhen sufficient memory exists in a system.vm.swappiness = 60 The default value.vm.swappiness = 100 The kernel will swap aggressively. Proxmox VE Administration Guide 39 / 3283.9 Certificate Management3.9.1 Certificates for communication within the clusterEach Proxmox VE cluster creates its own internal Certificate Authority (CA) and generates a self-signed cer-tificate for each node. These certificates are used for encrypted communication with the cluster’s pveproxyservice and the Shell/Console feature if SPICE is used.The CA certificate and key are stored in the pmxcfs (see the pmxcfs(8) manpage).3.9.2 Certificates for API and web GUIThe REST API and web GUI are provided by the pveproxy service, which runs on each node.You have the following options for the certificate used by pveproxy:1. By default the node-specific certificate in /etc/pve/nodes/NODENAME/pve-ssl.pem is used.This certificate is signed by the cluster CA and therefore not trusted by browsers and operating sys-tems by default.2. use an externally provided certificate (e.g. signed by a commercial CA).3. use ACME (e.g., Let’s Encrypt) to get a trusted certificate with automatic renewal.For options 2 and 3 the file /etc/pve/local/pveproxy-ssl.pem (and /etc/pve/local/pveproxy-ssl.key, which needs to be without password) is used.Certificates are managed with the Proxmox VE Node management command (see the pvenode(1) man-page).WarningDo not replace or manually modify the automatically generated node certificate files in /etc/pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key or the cluster CA filesin /etc/pve/pve-root-ca.pem and /etc/pve/priv/pve-root-ca.key.Getting trusted certificates via ACMEProxmox VE includes an implementation of the Automatic Certificate Management Environment ACME pro-tocol, allowing Proxmox VE admins to interface with Let’s Encrypt for easy setup of trusted TLS certificateswhich are accepted out of the box on most modern operating systems and browsers.Currently the two ACME endpoints implemented are Let’s Encrypt (LE) and its staging environment (seehttps://letsencrypt.org), both using the standalone HTTP challenge.Because of rate-limits you should use LE staging for experiments.There are a few prerequisites to use Let’s Encrypt:1. Port 80 of the node needs to be reachable from the internet.2. There must be no other listener on port 80. Proxmox VE Administration Guide 40 / 3283. The requested (sub)domain needs to resolve to a public IP of the Node.4. You have to accept the ToS of Let’s Encrypt.At the moment the GUI uses only the default ACME account.Example: Sample pvenode invocation for using Let’s Encrypt certificatesroot@proxmox:~# pvenode acme account register default mail@example.invalidDirectory endpoints:0) Let’s Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)1) Let’s Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/ ←-directory)2) CustomEnter selection:1Attempting to fetch Terms of Service from ’https://acme-staging-v02.api. ←-letsencrypt.org/directory’..Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November ←--15-2017.pdfDo you agree to the above terms? [y|N]yAttempting to register account with ’https://acme-staging-v02.api. ←-letsencrypt.org/directory’..Generating ACME account key..Registering ACME account..Registration successful, account URL: ’https://acme-staging-v02.api. ←-letsencrypt.org/acme/acct/xxxxxxx’Task OKroot@proxmox:~# pvenode acme account listdefaultroot@proxmox:~# pvenode config set --acme domains=example.invalidroot@proxmox:~# pvenode acme cert orderLoading ACME account detailsPlacing ACME orderOrder URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/ ←-xxxxxxxxxxxxxxGetting authorization details from’https://acme-staging-v02.api.letsencrypt.org/acme/authz/ ←-xxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxx-xxxxxxx’... pending!Setting up webserverTriggering validationSleeping for 5 secondsStatus is ’valid’!All domains validated!Creating CSRFinalizing order Proxmox VE Administration Guide 41 / 328Checking order statusvalid!Downloading certificateSetting pveproxy certificate and keyRestarting pveproxyTask OKAutomatic renewal of ACME certificatesIf a node has been successfully configured with an ACME-provided certificate (either via pvenode or via theGUI), the certificate will be automatically renewed by the pve-daily-update.service. Currently, renewal will beattempted if the certificate has expired or will expire in the next 30 days. Proxmox VE Administration Guide 42 / 328Chapter 4Hyper-converged InfrastructureProxmox VE is a virtualization platform that tightly integrates compute, storage and networking resources,manages highly available clusters, backup/restore as well as disaster recovery. All components are software-defined and compatible with one another.Therefore it is possible to administrate them like a single system via the centralized web management inter-face. These capabilities make Proxmox VE an ideal choice to deploy and manage an open source hyper-converged infrastructure.4.1 Benefits of a Hyper-Converged Infrastructure (HCI) with ProxmoxVEA hyper-converged infrastructure is especially useful for deployments in which a high infrastructure demandmeets a low administration budget, for distributed setups such as remote and branch office environments orfor virtual private and public clouds.HCI provides the following advantages:• Scalability: seamless expansion of compute, network and storage devices (i.e. scale up servers andstorage quickly and independently from each other).• Low cost: Proxmox VE is open source and integrates all components you need such as compute, storage,networking, backup, and management center. It can replace an expensive compute/storage infrastructure.• Data protection and efficiency: services such as backup and disaster recovery are integrated.• Simplicity: easy configuration and centralized administration.• Open Source: No vendor lock-in. Proxmox VE Administration Guide 43 / 3284.2 Manage Ceph Services on Proxmox VE NodesProxmox VE unifies your compute and storage systems, i.e. you can use the same physical nodes withina cluster for both computing (processing VMs and containers) and replicated storage. The traditional silosof compute and storage resources can be wrapped up into a single hyper-converged appliance. Separatestorage networks (SANs) and connections via network (NAS) disappear. With the integration of Ceph, anopen source software-defined storage platform, Proxmox VE has the ability to run and manage Ceph storagedirectly on the hypervisor nodes.Ceph is a distributed object store and file system designed to provide excellent performance, reliability andscalability.For small to mid sized deployments, it is possible to install a Ceph server for RADOS Block Devices (RBD)directly on your Proxmox VE cluster nodes, see Ceph RADOS Block Devices (RBD) Section 8.14. Recenthardware has plenty of CPU power and RAM, so running storage services and VMs on the same node ispossible.To simplify management, we provide pveceph - a tool to install and manage Ceph services on Proxmox VEnodes.Ceph consists of a couple of Daemons1, for use as a RBD storage:1Ceph intro http://docs.ceph.com/docs/master/start/intro/ Proxmox VE Administration Guide 44 / 328• Ceph Monitor (ceph-mon)• Ceph Manager (ceph-mgr)• Ceph OSD (ceph-osd; Object Storage Daemon)TipWe recommend to get familiar with the Ceph vocabulary.aaCeph glossary http://docs.ceph.com/docs/luminous/glossary4.2.1 PreconditionTo build a Proxmox Ceph Cluster there should be at least three (preferably) identical servers for the setup.A 10Gb network, exclusively used for Ceph, is recommended. A meshed network setup is also an option ifthere are no 10Gb switches available, see wiki .Check also the recommendations from Ceph’s website.4.2.2 Installation of Ceph PackagesOn each node run the installation script as follows:pveceph installThis sets up an apt package repository in /etc/apt/sources.list.d/ceph.list and installsthe required software. Proxmox VE Administration Guide 45 / 3284.2.3 Creating initial Ceph configurationAfter installation of packages, you need to create an initial Ceph configuration on just one node, based onyour network (10.10.10.0/24 in the following example) dedicated for Ceph:pveceph init --network 10.10.10.0/24This creates an initial config at /etc/pve/ceph.conf. That file is automatically distributed to all Prox-mox VE nodes by using pmxcfs Chapter 7. The command also creates a symbolic link from /etc/ceph/ceph.conf pointing to that file. So you can simply run Ceph commands without the need to specify aconfiguration file. Proxmox VE Administration Guide 46 / 3284.2.4 Creating Ceph MonitorsThe Ceph Monitor (MON)2maintains a master copy of the cluster map. For HA you need to have at least 3monitors.On each node where you want to place a monitor (three monitors are recommended), create it by using theCeph → Monitor tab in the GUI or run.pveceph createmonThis will also install the needed Ceph Manager (ceph-mgr) by default. If you do not want to install a manager,specify the -exclude-manager option.4.2.5 Creating Ceph ManagerThe Manager daemon runs alongside the monitors. It provides interfaces for monitoring the cluster. Sincethe Ceph luminous release the ceph-mgr3daemon is required. During monitor installation the ceph managerwill be installed as well.2Ceph Monitor http://docs.ceph.com/docs/luminous/start/intro/3Ceph Manager http://docs.ceph.com/docs/luminous/mgr/ Proxmox VE Administration Guide vi6 Cluster Manager 636.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.2 Preparing Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646.3 Create the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646.3.1 Multiple Clusters In Same Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646.4 Adding Nodes to the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656.4.1 Adding Nodes With Separated Cluster Network . . . . . . . . . . . . . . . . . . . . . 666.5 Remove a Cluster Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 666.5.1 Separate A Node Without Reinstalling . . . . . . . . . . . . . . . . . . . . . . . . . . 676.6 Quorum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696.7 Cluster Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696.7.1 Network Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696.7.2 Separate Cluster Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 706.7.3 Redundant Ring Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.7.4 RRP On Cluster Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.7.5 RRP On Existing Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.8 Corosync Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746.8.1 Edit corosync.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746.8.2 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756.8.3 Corosync Configuration Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766.9 Cluster Cold Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766.10 Guest Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.10.1 Migration Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.10.2 Migration Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 777 Proxmox Cluster File System (pmxcfs) 797.1 POSIX Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 797.2 File Access Rights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807.3 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807.4 File System Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807.4.1 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807.4.2 Symbolic links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817.4.3 Special status files for debugging (JSON) . . . . . . . . . . . . . . . . . . . . . . . . 817.4.4 Enable/Disable debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817.5 Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817.5.1 Remove Cluster configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817.5.2 Recovering/Moving Guests from Failed Nodes . . . . . . . . . . . . . . . . . . . . . . 82 Proxmox VE Administration Guide 47 / 328NoteIt is recommended to install the Ceph Manager on the monitor nodes. For high availability install morethen one manager.pveceph createmgr4.2.6 Creating Ceph OSDsvia GUI or via CLI as follows:pveceph createosd /dev/sd[X]TipWe recommend a Ceph cluster size, starting with 12 OSDs, distributed evenly among your, at least threenodes (4 OSDs on each node). Proxmox VE Administration Guide 48 / 328Ceph BluestoreStarting with the Ceph Kraken release, a new Ceph OSD storage type was introduced, the so called Blue-store4. In Ceph luminous this store is the default when creating OSDs.pveceph createosd /dev/sd[X]NoteIn order to select a disk in the GUI, to be more failsafe, the disk needs to have a GPTapartition table. Youcan create this with gdisk /dev/sd(x). If there is no GPT, you cannot select the disk as DB/WAL.aGPT partition table https://en.wikipedia.org/wiki/GUID_Partition_TableIf you want to use a separate DB/WAL device for your OSDs, you can specify it through the -wal_dev option.pveceph createosd /dev/sd[X] -wal_dev /dev/sd[Y]NoteThe DB stores BlueStore’s internal metadata and the WAL is BlueStore’s internal journal or write-aheadlog. It is recommended to use a fast SSDs or NVRAM for better performance.Ceph FilestoreTill Ceph luminous, Filestore was used as storage type for Ceph OSDs. It can still be used and might givebetter performance in small setups, when backed by a NVMe SSD or similar.pveceph createosd /dev/sd[X] -bluestore 0NoteIn order to select a disk in the GUI, the disk needs to have a GPTapartition table. You can create this withgdisk /dev/sd(x). If there is no GPT, you cannot select the disk as journal. Currently the journalsize is fixed to 5 GB.If you want to use a dedicated SSD journal disk:pveceph createosd /dev/sd[X] -journal_dev /dev/sd[Y] -bluestore 0Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD journal disk.pveceph createosd /dev/sdf -journal_dev /dev/sdb -bluestore 0This partitions the disk (data and journal partition), creates filesystems and starts the OSD, afterwards it isrunning and fully functional.4Ceph Bluestore http://ceph.com/community/new-luminous-bluestore/ Proxmox VE Administration Guide 49 / 328NoteThis command refuses to initialize disk when it detects existing data. So if you want to overwrite a diskyou should remove existing data first. You can do that using: ceph-disk zap /dev/sd[X]You can create OSDs containing both journal and data partitions or you can place the journal on a dedicatedSSD. Using a SSD journal disk is highly recommended to achieve good performance.4.2.7 Creating Ceph PoolsA pool is a logical group for storing objects. It holds Placement Groups (PG), a collection of objects.When no options are given, we set a default of 64 PGs, a size of 3 replicas and a min_size of 2 replicasfor serving objects in a degraded state.NoteThe default number of PGs works for 2-6 disks. Ceph throws a "HEALTH_WARNING" if you have too fewor too many PGs in your cluster. Proxmox VE Administration Guide 50 / 328It is advised to calculate the PG number depending on your setup, you can find the formula and the PGcalculator5online. While PGs can be increased later on, they can never be decreased.You can create pools through command line or on the GUI on each PVE host under Ceph → Pools.pveceph createpool <name>If you would like to automatically get also a storage definition for your pool, active the checkbox "Add stor-ages" on the GUI or use the command line option --add_storages on pool creation.Further information on Ceph pool handling can be found in the Ceph pool operation6manual.4.2.8 Ceph CRUSH & device classesThe foundation of Ceph is its algorithm, Controlled Replication Under Scalable Hashing (CRUSH7).CRUSH calculates where to store to and retrieve data from, this has the advantage that no central index ser-vice is needed. CRUSH works with a map of OSDs, buckets (device locations) and rulesets (data replication)for pools.NoteFurther information can be found in the Ceph documentation, under the section CRUSH mapa.aCRUSH map http://docs.ceph.com/docs/luminous/rados/operations/crush-map/This map can be altered to reflect different replication hierarchies. The object replicas can be separated (eg.failure domains), while maintaining the desired distribution.A common use case is to use different classes of disks for different Ceph pools. For this reason, Cephintroduced the device classes with luminous, to accommodate the need for easy ruleset generation.The device classes can be seen in the ceph osd tree output. These classes represent their own root bucket,which can be seen with the below command.ceph osd crush tree --show-shadowExample output form the above command:ID CLASS WEIGHT TYPE NAME-16 nvme 2.18307 root default~nvme-13 nvme 0.72769 host sumi1~nvme12 nvme 0.72769 osd.12-14 nvme 0.72769 host sumi2~nvme13 nvme 0.72769 osd.13-15 nvme 0.72769 host sumi3~nvme14 nvme 0.72769 osd.14-1 7.70544 root default-3 2.56848 host sumi112 nvme 0.72769 osd.12-5 2.56848 host sumi25PG calculator http://ceph.com/pgcalc/6Ceph pool operation http://docs.ceph.com/docs/luminous/rados/operations/pools/7CRUSH https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf Proxmox VE Administration Guide 51 / 32813 nvme 0.72769 osd.13-7 2.56848 host sumi314 nvme 0.72769 osd.14To let a pool distribute its objects only on a specific device class, you need to create a ruleset with the specificclass first.ceph osd crush rule create-replicated <rule-name> <root> <failure-domain> < ←-class><rule-name> name of the rule, to connect with a pool (seen in GUI & CLI)<root> which crush root it should belong to (default ceph root "default")<failure-domain> at which failure-domain the objects should be distributed (usually host)<class> what type of OSD backing store to use (eg. nvme, ssd, hdd)Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.ceph osd pool set <pool-name> crush_rule <rule-name>TipIf the pool already contains objects, all of these have to be moved accordingly. Depending on your setupthis may introduce a big performance hit on your cluster. As an alternative, you can create a new pool andmove disks separately. Proxmox VE Administration Guide 52 / 3284.2.9 Ceph ClientYou can then configure Proxmox VE to use such pools to store VM or Container images. Simply use the GUItoo add a new RBD storage (see section Ceph RADOS Block Devices (RBD) Section 8.14).You also need to copy the keyring to a predefined location for a external Ceph cluster. If Ceph is installed onthe Proxmox nodes itself, then this will be done automatically.NoteThe file name needs to be <storage_id> + `.keyring - <storage_id> is the expression afterrbd: in /etc/pve/storage.cfg which is my-ceph-storage in the following example:mkdir /etc/pve/priv/cephcp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage. ←-keyring Proxmox VE Administration Guide 53 / 328Chapter 5Graphical User InterfaceProxmox VE is simple. There is no need to install a separate management tool, and everything can be donethrough your web browser (Latest Firefox or Google Chrome is preferred). A built-in HTML5 console is usedto access the guest console. As an alternative, SPICE can be used.Because we use the Proxmox cluster file system (pmxcfs), you can connect to any node to manage theentire cluster. Each node can manage the entire cluster. There is no need for a dedicated manager node.You can use the web-based administration interface with any modern browser. When Proxmox VE detectsthat you are connecting from a mobile device, you are redirected to a simpler, touch-based user interface.The web interface can be reached via https://youripaddress:8006 (default login is: root, and the password isspecified during the installation process).5.1 Features• Seamless integration and management of Proxmox VE clusters• AJAX technologies for dynamic updates of resources• Secure access to all Virtual Machines and Containers via SSL encryption (https)• Fast search-driven interface, capable of handling hundreds and probably thousands of VMs• Secure HTML5 console or SPICE• Role based permission management for all objects (VMs, storages, nodes, etc.)• Support for multiple authentication sources (e.g. local, MS ADS, LDAP, . . . )• Two-Factor Authentication (OATH, Yubikey)• Based on ExtJS 6.x JavaScript framework Proxmox VE Administration Guide 54 / 3285.2 LoginWhen you connect to the server, you will first see the login window. Proxmox VE supports various authen-tication backends (Realm), and you can select the language here. The GUI is translated to more than 20languages.NoteYou can save the user name on the client side by selection the checkbox at the bottom. This saves sometyping when you login next time.5.3 GUI Overview Proxmox VE Administration Guide 55 / 328The Proxmox VE user interface consists of four regions.Header On top. Shows status information and contains buttons for most important actions.Resource Tree At the left side. A navigation tree where you can select specific objects.Content Panel Center region. Selected objects displays configuration options and status here.Log Panel At the bottom. Displays log entries for recent tasks. You can double-click on thoselog entries to get more details, or to abort a running task.NoteYou can shrink and expand the size of the resource tree and log panel, or completely hide the log panel.This can be helpful when you work on small displays and want more space to view other content.5.3.1 HeaderOn the top left side, the first thing you see is the Proxmox logo. Next to it is the current running version ofProxmox VE. In the search bar nearside you can search for specific objects (VMs, containers, nodes, . . . ).This is sometimes faster than selecting an object in the resource tree.To the right of the search bar we see the identity (login name). The gear symbol is a button opening theMy Settings dialog. There you can customize some client side user interface setting (reset the saved loginname, reset saved layout).The rightmost part of the header contains four buttons:Help Opens a new browser window showing the reference documentation.Create VM Opens the virtual machine creation wizard.Create CT Open the container creation wizard. Proxmox VE Administration Guide 56 / 328Logout Logout, and show the login dialog again.5.3.2 Resource TreeThis is the main navigation tree. On top of the tree you can select some predefined views, which changesthe structure of the tree below. The default view is Server View, and it shows the following object types:Datacenter Contains cluster wide setting (relevant for all nodes).Node Represents the hosts inside a cluster, where the guests runs.Guest VMs, Containers and Templates.Storage Data Storage.Pool It is possible to group guests using a pool to simplify management.The following view types are available:Server View Shows all kind of objects, grouped by nodes.Folder View Shows all kind of objects, grouped by object type.Storage View Only show storage objects, grouped by nodes.Pool View Show VMs and Containers, grouped by pool.5.3.3 Log PanelThe main purpose of the log panel is to show you what is currently going on in your cluster. Actions likecreating an new VM are executed in background, and we call such background job a task.Any output from such task is saved into a separate log file. You can view that log by simply double-click atask log entry. It is also possible to abort a running task there.Please note that we display most recent tasks from all cluster nodes here. So you can see when somebodyelse is working on another cluster node in real-time.NoteWe remove older and finished task from the log panel to keep that list short. But you can still find thosetasks in the Task History within the node panel.Some short running actions simply sends logs to all cluster members. You can see those messages in theCluster log panel. Proxmox VE Administration Guide vii8 Proxmox VE Storage 838.1 Storage Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 838.1.1 Thin Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 848.2 Storage Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 848.2.1 Storage Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 858.2.2 Common Storage Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 858.3 Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 868.3.1 Volume Ownership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878.4 Using the Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878.4.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878.5 Directory Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 888.5.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 898.5.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 898.5.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 908.5.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 908.6 NFS Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 918.6.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 918.6.2 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 928.6.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 928.7 CIFS Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 928.7.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 928.7.2 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 938.7.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948.8 GlusterFS Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948.8.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948.8.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958.8.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958.9 Local ZFS Pool Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958.9.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958.9.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 968.9.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 968.9.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 968.10 LVM Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 968.10.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 978.10.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Proxmox VE Administration Guide 57 / 3285.4 Content PanelsWhen you select something in the resource tree, the corresponding object displays configuration and statusinformation in the content panel. The following sections give a brief overview of the functionality. Please referto the individual chapters inside the reference documentation to get more detailed information.5.4.1 DatacenterOn the datacenter level you can access cluster wide settings and information.• Search: it is possible to search anything in cluster ,this can be a node, VM, Container, Storage or a pool.• Summary: gives a brief overview over the cluster health.• Options: can show and set defaults, which apply cluster wide.• Storage: is the place where a storage will add/managed/removed.• Backup: has the capability to schedule Backups. This is cluster wide, so you do not care about where theVM/Container are on your cluster at schedule time. Proxmox VE Administration Guide 58 / 328• Permissions: will manage user and group permission, LDAP, MS-AD and Two-Factor authentication canbe setup here.• HA: will manage the Proxmox VE High-Availability• Firewall: on this level the Proxmox Firewall works cluster wide and makes templates which are clusterwide available.• Support: here you get all information about your support subscription.If you like to have more information about this see the corresponding chapter.5.4.2 NodesAll belongs of a node can be managed at this level.• Search: it is possible to search anything on the node, this can be a VM, Container, Storage or a pool.• Summary: gives a brief overview over the resource usage.• Shell: log you in the shell of the node. Proxmox VE Administration Guide 59 / 328• System: is for configuring the network, dns and time, and also shows your syslog.• Updates: will upgrade the system and informs you about new packets.• Firewall: on this level is only for this node.• Disk: gives you an brief overview about you physical hard drives and how they are used.• Ceph: is only used if you have installed a Ceph sever on you host. Then you can manage your Cephcluster and see the status of it here.• Task History: here all past task are shown.• Subscription: here you can upload you subscription key and get a system overview in case of a supportcase.5.4.3 GuestsThere are two differed kinds of VM types and both types can be converted to a template. One of them areKernel-based Virtual Machine (KVM) and the other one are Linux Containers (LXC). General the navigationare the same only some option are different.In the main management center the VM navigation begin if a VM is selected in the left tree.The top header contains important VM operation commands like Start, Shutdown, Reset, Remove, Migrate,Console and Help. Some of them have hidden buttons like Shutdown has Stop and Console contains thedifferent console types SPICE, noVNC and xterm.js.On the right side the content switch white the focus of the option.On the left side. All available options are listed one below the other. Proxmox VE Administration Guide 60 / 328• Summary: gives a brief overview over the VM activity.• Console: an interactive console to your VM.• (KVM)Hardware: shows and set the Hardware of the KVM VM.• (LXC)Resources: defines the LXC Hardware opportunities.• (LXC)Network: the LXC Network settings.• (LXC)DNS: the LXC DNS settings.• Options: all VM options can be set here, this distinguishes between KVM and LXC.• Task History: here all previous task from this VM will be shown.• (KVM) Monitor: is the interactive communication interface to the KVM process.• Backup: shows the available backups from this VM and also create a backupset.• Replication: shows the replication jobs for this VM and allows to create new jobs.• Snapshots: manage VM snapshots.• Firewall: manage the firewall on VM level.• Permissions: manage the user permission for this VM. Proxmox VE Administration Guide 61 / 3285.4.4 StorageIn this view we have a two partition split view. On the left side we have the storage options and on the rightside the content of the selected option will shown.• Summary: show you important information about your storage like Usage, Type, Content, Active andEnabled.• Content: Here all contend will listed grouped by content.• Permissions: manage the user permission for this storage. Proxmox VE Administration Guide 62 / 3285.4.5 PoolsIn this view we have a two partition split view. On the left side we have the logical pool options and on theright side the content of the selected option will shown.• Summary: show the description of the pool.• Members: Here all members of this pool will listed and can be managed.• Permissions: manage the user permission for this pool. Proxmox VE Administration Guide 63 / 328Chapter 6Cluster ManagerThe Proxmox VE cluster manager pvecm is a tool to create a group of physical servers. Such a group iscalled a cluster. We use the Corosync Cluster Engine for reliable group communication, and such clusterscan consist of up to 32 physical nodes (probably more, dependent on network latency).pvecm can be used to create a new cluster, join nodes to a cluster, leave the cluster, get status informa-tion and do various other cluster related tasks. The Proxmox Cluster File System (“pmxcfs”) is used totransparently distribute the cluster configuration to all cluster nodes.Grouping nodes into a cluster has the following advantages:• Centralized, web based management• Multi-master clusters: each node can do all management task• pmxcfs: database-driven file system for storing configuration files, replicated in real-time on all nodesusing corosync.• Easy migration of virtual machines and containers between physical hosts• Fast deployment• Cluster-wide services like firewall and HA6.1 Requirements• All nodes must be in the same network as corosync uses IP Multicast to communicate between nodes(also see Corosync Cluster Engine). Corosync uses UDP ports 5404 and 5405 for cluster communication.NoteSome switches do not support IP multicast by default and must be manually enabled first.• Date and time have to be synchronized.• SSH tunnel on TCP port 22 between nodes is used. Proxmox VE Administration Guide 64 / 328• If you are interested in High Availability, you need to have at least three nodes for reliable quorum. Allnodes should have the same version.• We recommend a dedicated NIC for the cluster traffic, especially if you use shared storage.NoteIt is not possible to mix Proxmox VE 3.x and earlier with Proxmox VE 4.0 cluster nodes.6.2 Preparing NodesFirst, install Proxmox VE on all nodes. Make sure that each node is installed with the final hostname and IPconfiguration. Changing the hostname and IP is not possible after cluster creation.Currently the cluster creation has to be done on the console, so you need to login via ssh.6.3 Create the ClusterLogin via ssh to the first Proxmox VE node. Use a unique name for your cluster. This name cannot bechanged later.hp1# pvecm create YOUR-CLUSTER-NAMECautionThe cluster name is used to compute the default multicast address. Please use unique clusternames if you run more than one cluster inside your network.To check the state of your cluster use:hp1# pvecm status6.3.1 Multiple Clusters In Same NetworkIt is possible to create multiple clusters in the same physical or logical network. Each cluster must havea unique name, which is used to generate the cluster’s multicast group address. As long as no duplicatecluster names are configured in one network segment, the different clusters won’t interfere with each other.If multiple clusters operate in a single network it may be beneficial to setup an IGMP querier and enableIGMP Snooping in said network. This may reduce the load of the network significantly because multicastpackets are only delivered to endpoints of the respective member nodes. Proxmox VE Administration Guide 65 / 3286.4 Adding Nodes to the ClusterLogin via ssh to the node you want to add.hp2# pvecm add IP-ADDRESS-CLUSTERFor IP-ADDRESS-CLUSTER use the IP from an existing cluster node.CautionA new node cannot hold any VMs, because you would get conflicts about identical VM IDs. Also,all existing configuration in /etc/pve is overwritten when you join a new node to the cluster. Toworkaround, use vzdump to backup and restore to a different VMID after adding the node to thecluster.To check the state of cluster:# pvecm statusCluster status after adding 4 nodeshp2# pvecm statusQuorum information~~~~~~~~~~~~~~~~~~Date: Mon Apr 20 12:30:13 2015Quorum provider: corosync_votequorumNodes: 4Node ID: 0x00000001Ring ID: 1928Quorate: YesVotequorum information~~~~~~~~~~~~~~~~~~~~~~Expected votes: 4Highest expected: 4Total votes: 4Quorum: 2Flags: QuorateMembership information~~~~~~~~~~~~~~~~~~~~~~Nodeid Votes Name0x00000001 1 192.168.15.910x00000002 1 192.168.15.92 (local)0x00000003 1 192.168.15.930x00000004 1 192.168.15.94If you only want the list of all nodes use:# pvecm nodes Proxmox VE Administration Guide 66 / 328List nodes in a clusterhp2# pvecm nodesMembership information~~~~~~~~~~~~~~~~~~~~~~Nodeid Votes Name1 1 hp12 1 hp2 (local)3 1 hp34 1 hp46.4.1 Adding Nodes With Separated Cluster NetworkWhen adding a node to a cluster with a separated cluster network you need to use the ringX_addr parame-ters to set the nodes address on those networks:pvecm add IP-ADDRESS-CLUSTER -ring0_addr IP-ADDRESS-RING0If you want to use the Redundant Ring Protocol you will also want to pass the ring1_addr parameter.6.5 Remove a Cluster NodeCautionRead carefully the procedure before proceeding, as it could not be what you want or need.Move all virtual machines from the node. Make sure you have no local data or backups you want to keep, orsave them accordingly. In the following example we will remove the node hp4 from the cluster.Log in to a different cluster node (not hp4), and issue a pvecm nodes command to identify the node IDto remove:hp1# pvecm nodesMembership information~~~~~~~~~~~~~~~~~~~~~~Nodeid Votes Name1 1 hp1 (local)2 1 hp23 1 hp34 1 hp4At this point you must power off hp4 and make sure that it will not power on again (in the network) as it is.ImportantAs said above, it is critical to power off the node before removal, and make sure that it will neverpower on again (in the existing cluster network) as it is. If you power on the node as it is, yourcluster will be screwed up and it could be difficult to restore a clean cluster state. Proxmox VE Administration Guide viii8.10.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 978.10.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 988.11 LVM thin Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 988.11.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 988.11.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998.11.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998.11.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998.12 Open-iSCSI initiator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998.12.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998.12.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1008.12.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1008.12.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1008.13 User Mode iSCSI Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1018.13.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1018.13.2 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1018.14 Ceph RADOS Block Devices (RBD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1018.14.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1028.14.2 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1038.14.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1039 Storage Replication 1049.1 Supported Storage Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1049.2 Schedule Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1059.2.1 Detailed Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1059.2.2 Examples: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1069.3 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1069.3.1 Possible issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1069.3.2 Migrating a guest in case of Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1079.3.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1079.4 Managing Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1089.5 Command Line Interface Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10810 Qemu/KVM Virtual Machines 10910.1 Emulated devices and paravirtualized devices . . . . . . . . . . . . . . . . . . . . . . . . . . 10910.2 Virtual Machines Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11010.2.1 General Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Proxmox VE Administration Guide 67 / 328After powering off the node hp4, we can safely remove it from the cluster.hp1# pvecm delnode hp4If the operation succeeds no output is returned, just check the node list again with pvecm nodes or pvecmstatus. You should see something like:hp1# pvecm statusQuorum information~~~~~~~~~~~~~~~~~~Date: Mon Apr 20 12:44:28 2015Quorum provider: corosync_votequorumNodes: 3Node ID: 0x00000001Ring ID: 1992Quorate: YesVotequorum information~~~~~~~~~~~~~~~~~~~~~~Expected votes: 3Highest expected: 3Total votes: 3Quorum: 3Flags: QuorateMembership information~~~~~~~~~~~~~~~~~~~~~~Nodeid Votes Name0x00000001 1 192.168.15.90 (local)0x00000002 1 192.168.15.910x00000003 1 192.168.15.92If, for whatever reason, you want that this server joins the same cluster again, you have to• reinstall Proxmox VE on it from scratch• then join it, as explained in the previous section.6.5.1 Separate A Node Without ReinstallingCautionThis is not the recommended method, proceed with caution. Use the above mentioned method ifyou’re unsure.You can also separate a node from a cluster without reinstalling it from scratch. But after removing thenode from the cluster it will still have access to the shared storages! This must be resolved before you startremoving the node from the cluster. A Proxmox VE cluster cannot share the exact same storage with anothercluster, as storage locking doesn’t work over cluster boundary. Further, it may also lead to VMID conflicts. Proxmox VE Administration Guide 68 / 328Its suggested that you create a new storage where only the node which you want to separate has access.This can be an new export on your NFS or a new Ceph pool, to name a few examples. Its just important thatthe exact same storage does not gets accessed by multiple clusters. After setting this storage up move alldata from the node and its VMs to it. Then you are ready to separate the node from the cluster.WarningEnsure all shared resources are cleanly separated! You will run into conflicts and problems else.First stop the corosync and the pve-cluster services on the node:systemctl stop pve-clustersystemctl stop corosyncStart the cluster filesystem again in local mode:pmxcfs -lDelete the corosync configuration files:rm /etc/pve/corosync.confrm /etc/corosync/*You can now start the filesystem again as normal service:killall pmxcfssystemctl start pve-clusterThe node is now separated from the cluster. You can deleted it from a remaining node of the cluster with:pvecm delnode oldnodeIf the command failed, because the remaining node in the cluster lost quorum when the now separate nodeexited, you may set the expected votes to 1 as a workaround:pvecm expected 1And the repeat the pvecm delnode command.Now switch back to the separated node, here delete all remaining files left from the old cluster. This ensuresthat the node can be added to another cluster again without problems.rm /var/lib/corosync/*As the configuration files from the other nodes are still in the cluster filesystem you may want to clean thoseup too. Remove simply the whole directory recursive from /etc/pve/nodes/NODENAME, but check threetimes that you used the correct one before deleting it.CautionThe nodes SSH keys are still in the authorized_key file, this means the nodes can still connect toeach other with public key authentication. This should be fixed by removing the respective keysfrom the /etc/pve/priv/authorized_keys file. Proxmox VE Administration Guide 69 / 3286.6 QuorumProxmox VE use a quorum-based technique to provide a consistent state among all cluster nodes.A quorum is the minimum number of votes that a distributed transaction has to obtain in orderto be allowed to perform an operation in a distributed system.— from Wikipedia Quorum (distributed computing)In case of network partitioning, state changes requires that a majority of nodes are online. The clusterswitches to read-only mode if it loses quorum.NoteProxmox VE assigns a single vote to each node by default.6.7 Cluster NetworkThe cluster network is the core of a cluster. All messages sent over it have to be delivered reliable to allnodes in their respective order. In Proxmox VE this part is done by corosync, an implementation of a highperformance low overhead high availability development toolkit. It serves our decentralized configuration filesystem (pmxcfs).6.7.1 Network RequirementsThis needs a reliable network with latencies under 2 milliseconds (LAN performance) to work properly. Whilecorosync can also use unicast for communication between nodes its highly recommended to have a multi-cast capable network. The network should not be used heavily by other members, ideally corosync runs onits own network. never share it with network where storage communicates too.Before setting up a cluster it is good practice to check if the network is fit for that purpose.• Ensure that all nodes are in the same subnet. This must only be true for the network interfaces used forcluster communication (corosync).• Ensure all nodes can reach each other over those interfaces, using ping is enough for a basic test.• Ensure that multicast works in general and a high package rates. This can be done with the omping tool.The final "%loss" number should be < 1%.omping -c 10000 -i 0.001 -F -q NODE1-IP NODE2-IP ...• Ensure that multicast communication works over an extended period of time. This uncovers problemswhere IGMP snooping is activated on the network but no multicast querier is active. This test has aduration of around 10 minutes.omping -c 600 -i 1 -q NODE1-IP NODE2-IP ...Your network is not ready for clustering if any of these test fails. Recheck your network configuration. Es-pecially switches are notorious for having multicast disabled by default or IGMP snooping enabled with noIGMP querier active.In smaller cluster its also an option to use unicast if you really cannot get multicast to work. Proxmox VE Administration Guide 70 / 3286.7.2 Separate Cluster NetworkWhen creating a cluster without any parameters the cluster network is generally shared with the Web UI andthe VMs and its traffic. Depending on your setup even storage traffic may get sent over the same network.Its recommended to change that, as corosync is a time critical real time application.Setting Up A New NetworkFirst you have to setup a new network interface. It should be on a physical separate network. Ensure thatyour network fulfills the cluster network requirements.Separate On Cluster CreationThis is possible through the ring0_addr and bindnet0_addr parameter of the pvecm create command usedfor creating a new cluster.If you have setup an additional NIC with a static address on 10.10.10.1/25 and want to send and receive allcluster communication over this interface you would execute:pvecm create test --ring0_addr 10.10.10.1 --bindnet0_addr 10.10.10.0To check if everything is working properly execute:systemctl status corosyncAfterwards, proceed as descripted in the section to add nodes with a separated cluster network.Separate After Cluster CreationYou can do this also if you have already created a cluster and want to switch its communication to anothernetwork, without rebuilding the whole cluster. This change may lead to short durations of quorum loss in thecluster, as nodes have to restart corosync and come up one after the other on the new network.Check how to edit the corosync.conf file first. The open it and you should see a file similar to:logging {debug: offto_syslog: yes}nodelist {node {name: duenodeid: 2quorum_votes: 1ring0_addr: due}node {name: trenodeid: 3 Proxmox VE Administration Guide 71 / 328quorum_votes: 1ring0_addr: tre}node {name: unonodeid: 1quorum_votes: 1ring0_addr: uno}}quorum {provider: corosync_votequorum}totem {cluster_name: thomas-testclusterconfig_version: 3ip_version: ipv4secauth: onversion: 2interface {bindnetaddr: 192.168.30.50ringnumber: 0}}The first you want to do is add the name properties in the node entries if you do not see them already. Thosemust match the node name.Then replace the address from the ring0_addr properties with the new addresses. You may use plain IPaddresses or also hostnames here. If you use hostnames ensure that they are resolvable from all nodes.In my example I want to switch my cluster communication to the 10.10.10.1/25 network. So I replace allring0_addr respectively. I also set the bindnetaddr in the totem section of the config to an address of thenew network. It can be any address from the subnet configured on the new network interface.After you increased the config_version property the new configuration file should look like:logging {debug: offto_syslog: yes}nodelist {node {name: duenodeid: 2quorum_votes: 1ring0_addr: 10.10.10.2 Proxmox VE Administration Guide 72 / 328}node {name: trenodeid: 3quorum_votes: 1ring0_addr: 10.10.10.3}node {name: unonodeid: 1quorum_votes: 1ring0_addr: 10.10.10.1}}quorum {provider: corosync_votequorum}totem {cluster_name: thomas-testclusterconfig_version: 4ip_version: ipv4secauth: onversion: 2interface {bindnetaddr: 10.10.10.1ringnumber: 0}}Now after a final check whether all changed information is correct we save it and see again the edit corosync.conffile section to learn how to bring it in effect.As our change cannot be enforced live from corosync we have to do an restart.On a single node execute:systemctl restart corosyncNow check if everything is fine:systemctl status corosyncIf corosync runs again correct restart corosync also on all other nodes. They will then join the clustermembership one by one on the new network. Proxmox VE Administration Guide 73 / 3286.7.3 Redundant Ring ProtocolTo avoid a single point of failure you should implement counter measurements. This can be on the hardwareand operating system level through network bonding.Corosync itself offers also a possibility to add redundancy through the so called Redundant Ring Protocol.This protocol allows running a second totem ring on another network, this network should be physicallyseparated from the other rings network to actually increase availability.6.7.4 RRP On Cluster CreationThe pvecm create command provides the additional parameters bindnetX_addr, ringX_addr and rrp_mode,can be used for RRP configuration.NoteSee the glossary if you do not know what each parameter means.So if you have two networks, one on the 10.10.10.1/24 and the other on the 10.10.20.1/24 subnet you wouldexecute:pvecm create CLUSTERNAME -bindnet0_addr 10.10.10.1 -ring0_addr 10.10.10.1 \-bindnet1_addr 10.10.20.1 -ring1_addr 10.10.20.16.7.5 RRP On Existing ClustersYou will take similar steps as described in separating the cluster network to enable RRP on an alreadyrunning cluster. The single difference is, that you will add ring1 and use it instead of ring0.First add a new interface subsection in the totem section, set its ringnumber property to 1. Setthe interfaces bindnetaddr property to an address of the subnet you have configured for your new ring.Further set the rrp_mode to passive, this is the only stable mode.Then add to each node entry in the nodelist section its new ring1_addr property with the nodesadditional ring address.So if you have two networks, one on the 10.10.10.1/24 and the other on the 10.10.20.1/24 subnet, the finalconfiguration file should look like:totem {cluster_name: tweakconfig_version: 9ip_version: ipv4rrp_mode: passivesecauth: onversion: 2interface {bindnetaddr: 10.10.10.1ringnumber: 0}interface {bindnetaddr: 10.10.20.1 Proxmox VE Administration Guide 74 / 328ringnumber: 1}}nodelist {node {name: pvecm1nodeid: 1quorum_votes: 1ring0_addr: 10.10.10.1ring1_addr: 10.10.20.1}node {name: pvecm2nodeid: 2quorum_votes: 1ring0_addr: 10.10.10.2ring1_addr: 10.10.20.2}[...] # other cluster nodes here}[...] # other remaining config sections hereBring it in effect like described in the edit the corosync.conf file section.This is a change which cannot take live in effect and needs at least a restart of corosync. Recommended isa restart of the whole cluster.If you cannot reboot the whole cluster ensure no High Availability services are configured and the stop thecorosync service on all nodes. After corosync is stopped on all nodes start it one after the other again.6.8 Corosync ConfigurationThe /etc/pve/corosync.conf file plays a central role in Proxmox VE cluster. It controls the clustermember ship and its network. For reading more about it check the corosync.conf man page:man corosync.confFor node membership you should always use the pvecm tool provided by Proxmox VE. You may have toedit the configuration file manually for other changes. Here are a few best practice tips for doing this.6.8.1 Edit corosync.confEditing the corosync.conf file can be not always straight forward. There are two on each cluster, one in /etc/pve/corosync.conf and the other in /etc/corosync/corosync.conf. Editing the onein our cluster file system will propagate the changes to the local one, but not vice versa. Proxmox VE Administration Guide 75 / 328The configuration will get updated automatically as soon as the file changes. This means changes whichcan be integrated in a running corosync will take instantly effect. So you should always make a copy and editthat instead, to avoid triggering some unwanted changes by an in between safe.cp /etc/pve/corosync.conf /etc/pve/corosync.conf.newThen open the Config file with your favorite editor, nano and vim.tiny are preinstalled on Proxmox VEfor example.NoteAlways increment the config_version number on configuration changes, omitting this can lead to problems.After making the necessary changes create another copy of the current working configuration file. Thisserves as a backup if the new configuration fails to apply or makes problems in other ways.cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bakThen move the new configuration file over the old one:mv /etc/pve/corosync.conf.new /etc/pve/corosync.confYou may check with the commandssystemctl status corosyncjournalctl -b -u corosyncIf the change could applied automatically. If not you may have to restart the corosync service via:systemctl restart corosyncOn errors check the troubleshooting section below.6.8.2 TroubleshootingIssue: quorum.expected_votes must be configuredWhen corosync starts to fail and you get the following message in the system log:[...]corosync[1647]: [QUORUM] Quorum provider: corosync_votequorum failed to ←-initialize.corosync[1647]: [SERV ] Service engine ’corosync_quorum’ failed to load ←-for reason’configuration error: nodelist or quorum.expected_votes must be ←-configured!’[...]It means that the hostname you set for corosync ringX_addr in the configuration could not be resolved. Proxmox VE Administration Guide 76 / 328Write Configuration When Not QuorateIf you need to change /etc/pve/corosync.conf on an node with no quorum, and you know what you do, use:pvecm expected 1This sets the expected vote count to 1 and makes the cluster quorate. You can now fix your configuration, orrevert it back to the last working backup.This is not enough if corosync cannot start anymore. Here its best to edit the local copy of the corosyncconfiguration in /etc/corosync/corosync.conf so that corosync can start again. Ensure that on all nodes thisconfiguration has the same content to avoid split brains. If you are not sure what went wrong it’s best to askthe Proxmox Community to help you.6.8.3 Corosync Configuration GlossaryringX_addrThis names the different ring addresses for the corosync totem rings used for the cluster communica-tion.bindnetaddrDefines to which interface the ring should bind to. It may be any address of the subnet configured onthe interface we want to use. In general its the recommended to just use an address a node uses onthis interface.rrp_modeSpecifies the mode of the redundant ring protocol and may be passive, active or none. Note that useof active is highly experimental and not official supported. Passive is the preferred mode, it may doublethe cluster communication throughput and increases availability.6.9 Cluster Cold StartIt is obvious that a cluster is not quorate when all nodes are offline. This is a common case after a powerfailure.NoteIt is always a good idea to use an uninterruptible power supply (“UPS”, also called “battery backup”) toavoid this state, especially if you want HA.On node startup, the pve-guests service is started and waits for quorum. Once quorate, it starts allguests which have the onboot flag set.When you turn on nodes, or when power comes back after power failure, it is likely that some nodes bootsfaster than others. Please keep in mind that guest startup is delayed until you reach quorum. Proxmox VE Administration Guide ix10.2.2 OS Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11110.2.3 Hard Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11110.2.4 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11310.2.5 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11610.2.6 Network Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11810.2.7 USB Passthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11910.2.8 BIOS and UEFI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12010.2.9 Automatic Start and Shutdown of Virtual Machines . . . . . . . . . . . . . . . . . . . 12010.3 Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12110.3.1 Online Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12110.3.2 Offline Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12210.4 Copies and Clones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12210.5 Virtual Machine Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12310.6 Importing Virtual Machines and disk images . . . . . . . . . . . . . . . . . . . . . . . . . . . 12410.6.1 Step-by-step example of a Windows OVF import . . . . . . . . . . . . . . . . . . . . . 12410.6.2 Adding an external disk image to a Virtual Machine . . . . . . . . . . . . . . . . . . . 12510.7 Cloud-Init Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12510.7.1 Preparing Cloud-Init Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12610.7.2 Deploying Cloud-Init Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12710.7.3 Cloud-Init specific Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12810.8 Managing Virtual Machines with qm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12910.8.1 CLI Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12910.9 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12910.9.1 File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13010.9.2 Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13010.9.3 Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13110.10Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15111 Proxmox Container Toolkit 15211.1 Technology Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15211.2 Security Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15311.3 Guest Operating System Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15311.4 Container Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15511.5 Container Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15611.5.1 FUSE Mounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15611.5.2 Using Quotas Inside Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Proxmox VE Administration Guide 77 / 3286.10 Guest MigrationMigrating virtual guests to other nodes is a useful feature in a cluster. There are settings to control thebehavior of such migrations. This can be done via the configuration file datacenter.cfg or for a specificmigration via API or command line parameters.It makes a difference if a Guest is online or offline, or if it has local resources (like a local disk).For Details about Virtual Machine Migration see the QEMU/KVM Migration Chapter Section 10.3For Details about Container Migration see the Container Migration Chapter Section 11.96.10.1 Migration TypeThe migration type defines if the migration data should be sent over an encrypted (secure) channel or anunencrypted (insecure) one. Setting the migration type to insecure means that the RAM content of avirtual guest gets also transferred unencrypted, which can lead to information disclosure of critical data frominside the guest (for example passwords or encryption keys).Therefore, we strongly recommend using the secure channel if you do not have full control over the networkand can not guarantee that no one is eavesdropping to it.NoteStorage migration does not follow this setting. Currently, it always sends the storage content over a securechannel.Encryption requires a lot of computing power, so this setting is often changed to "unsafe" to achieve betterperformance. The impact on modern systems is lower because they implement AES encryption in hardware.The performance impact is particularly evident in fast networks where you can transfer 10 Gbps or more.6.10.2 Migration NetworkBy default, Proxmox VE uses the network in which cluster communication takes place to send the migrationtraffic. This is not optimal because sensitive cluster traffic can be disrupted and this network may not havethe best bandwidth available on the node.Setting the migration network parameter allows the use of a dedicated network for the entire migration traffic.In addition to the memory, this also affects the storage traffic for offline migrations.The migration network is set as a network in the CIDR notation. This has the advantage that you do not haveto set individual IP addresses for each node. Proxmox VE can determine the real address on the destinationnode from the network specified in the CIDR form. To enable this, the network must be specified so thateach node has one, but only one IP in the respective network.ExampleWe assume that we have a three-node setup with three separate networks. One for public communicationwith the Internet, one for cluster communication and a very fast one, which we want to use as a dedicatednetwork for migration.A network configuration for such a setup might look as follows: Proxmox VE Administration Guide 78 / 328iface eno1 inet manual# public networkauto vmbr0iface vmbr0 inet staticaddress 192.X.Y.57netmask 255.255.250.0gateway 192.X.Y.1bridge_ports eno1bridge_stp offbridge_fd 0# cluster networkauto eno2iface eno2 inet staticaddress 10.1.1.1netmask 255.255.255.0# fast networkauto eno3iface eno3 inet staticaddress 10.1.2.1netmask 255.255.255.0Here, we will use the network 10.1.2.0/24 as a migration network. For a single migration, you can do thisusing the migration_network parameter of the command line tool:# qm migrate 106 tre --online --migration_network 10.1.2.0/24To configure this as the default network for all migrations in the cluster, set the migration property of the/etc/pve/datacenter.cfg file:# use dedicated migration networkmigration: secure,network=10.1.2.0/24NoteThe migration type must always be set when the migration network gets set in /etc/pve/datacenter.cfg. Proxmox VE Administration Guide 79 / 328Chapter 7Proxmox Cluster File System (pmxcfs)The Proxmox Cluster file system (“pmxcfs”) is a database-driven file system for storing configuration files,replicated in real time to all cluster nodes using corosync. We use this to store all PVE related configura-tion files.Although the file system stores all data inside a persistent database on disk, a copy of the data resides inRAM. That imposes restriction on the maximum size, which is currently 30MB. This is still enough to storethe configuration of several thousand virtual machines.This system provides the following advantages:• seamless replication of all configuration to all nodes in real time• provides strong consistency checks to avoid duplicate VM IDs• read-only when a node loses quorum• automatic updates of the corosync cluster configuration to all nodes• includes a distributed locking mechanism7.1 POSIX CompatibilityThe file system is based on FUSE, so the behavior is POSIX like. But some feature are simply not imple-mented, because we do not need them:• you can just generate normal files and directories, but no symbolic links, . . .• you can’t rename non-empty directories (because this makes it easier to guarantee that VMIDs are unique).• you can’t change file permissions (permissions are based on path)• O_EXCL creates were not atomic (like old NFS)• O_TRUNC creates are not atomic (FUSE restriction) Proxmox VE Administration Guide 80 / 3287.2 File Access RightsAll files and directories are owned by user root and have group www-data. Only root has write permis-sions, but group www-data can read most files. Files below the following paths:/etc/pve/priv//etc/pve/nodes/${NAME}/priv/are only accessible by root.7.3 TechnologyWe use the Corosync Cluster Engine for cluster communication, and SQlite for the database file. The filesystem is implemented in user space using FUSE.7.4 File System LayoutThe file system is mounted at:/etc/pve7.4.1 Filescorosync.conf Corosync cluster configuration file (previous toProxmox VE 4.x this file was called cluster.conf)storage.cfg Proxmox VE storage configurationdatacenter.cfg Proxmox VE datacenter wide configuration(keyboard layout, proxy, . . . )user.cfg Proxmox VE access control configuration(users/groups/. . . )domains.cfg Proxmox VE authentication domainsstatus.cfg Proxmox VE external metrics server configurationauthkey.pub Public key used by ticket systempve-root-ca.pem Public certificate of cluster CApriv/shadow.cfg Shadow password filepriv/authkey.key Private key used by ticket systempriv/pve-root-ca.key Private key of cluster CAnodes/<NAME>/pve-ssl.pem Public SSL certificate for web server (signed bycluster CA)nodes/<NAME>/pve-ssl.key Private SSL key for pve-ssl.pemnodes/<NAME>/pveproxy-ssl.pem Public SSL certificate (chain) for web server(optional override for pve-ssl.pem)nodes/<NAME>/pveproxy-ssl.key Private SSL key for pveproxy-ssl.pem(optional) Proxmox VE Administration Guide 81 / 328nodes/<NAME>/qemu-server/<VMID>.confVM configuration data for KVM VMsnodes/<NAME>/lxc/<VMID>.conf VM configuration data for LXC containersfirewall/cluster.fw Firewall configuration applied to all nodesfirewall/<NAME>.fw Firewall configuration for individual nodesfirewall/<VMID>.fw Firewall configuration for VMs and Containers7.4.2 Symbolic linkslocal nodes/<LOCAL_HOST_NAME>qemu-server nodes/<LOCAL_HOST_NAME>/qemu-server/lxc nodes/<LOCAL_HOST_NAME>/lxc/7.4.3 Special status files for debugging (JSON).version File versions (to detect file modifications).members Info about cluster members.vmlist List of all VMs.clusterlog Cluster log (last 50 entries).rrd RRD data (most recent entries)7.4.4 Enable/Disable debuggingYou can enable verbose syslog messages with:echo "1" >/etc/pve/.debugAnd disable verbose syslog messages with:echo "0" >/etc/pve/.debug7.5 RecoveryIf you have major problems with your Proxmox VE host, e.g. hardware issues, it could be helpful to justcopy the pmxcfs database file /var/lib/pve-cluster/config.db and move it to a new ProxmoxVE host. On the new host (with nothing running), you need to stop the pve-cluster service and replacethe config.db file (needed permissions 0600). Second, adapt /etc/hostname and /etc/hostsaccording to the lost Proxmox VE host, then reboot and check. (And don’t forget your VM/CT data)7.5.1 Remove Cluster configurationThe recommended way is to reinstall the node after you removed it from your cluster. This makes sure thatall secret cluster/ssh keys and any shared configuration data is destroyed.In some cases, you might prefer to put a node back to local mode without reinstall, which is described inSeparate A Node Without Reinstalling Proxmox VE Administration Guide 82 / 3287.5.2 Recovering/Moving Guests from Failed NodesFor the guest configuration files in nodes/<NAME>/qemu-server/ (VMs) and nodes/<NAME>/lxc/ (containers), Proxmox VE sees the containing node <NAME> as owner of the respective guest. Thisconcept enables the usage of local locks instead of expensive cluster-wide locks for preventing concurrentguest configuration changes.As a consequence, if the owning node of a guest fails (e.g., because of a power outage, fencing event, ..), aregular migration is not possible (even if all the disks are located on shared storage) because such a local lockon the (dead) owning node is unobtainable. This is not a problem for HA-managed guests, as Proxmox VE’sHigh Availability stack includes the necessary (cluster-wide) locking and watchdog functionality to ensurecorrect and automatic recovery of guests from fenced nodes.If a non-HA-managed guest has only shared disks (and no other local resources which are only availableon the failed node are configured), a manual recovery is possible by simply moving the guest configurationfile from the failed node’s directory in /etc/pve/ to an alive node’s directory (which changes the logicalowner or location of the guest).For example, recovering the VM with ID 100 from a dead node1 to another node node2 works with thefollowing command executed when logged in as root on any member node of the cluster:mv /etc/pve/nodes/node1/qemu-server/100.conf /etc/pve/nodes/node2/WarningBefore manually recovering a guest like this, make absolutely sure that the failed source nodeis really powered off/fenced. Otherwise Proxmox VE’s locking principles are violated by the mvcommand, which can have unexpected consequences.WarningGuest with local disks (or other local resources which are only available on the dead node) are notrecoverable like this. Either wait for the failed node to rejoin the cluster or restore such guests frombackups. Proxmox VE Administration Guide 83 / 328Chapter 8Proxmox VE StorageThe Proxmox VE storage model is very flexible. Virtual machine images can either be stored on one orseveral local storages, or on shared storage like NFS or iSCSI (NAS, SAN). There are no limits, and youmay configure as many storage pools as you like. You can use all storage technologies available for DebianLinux.One major benefit of storing VMs on shared storage is the ability to live-migrate running machines withoutany downtime, as all nodes in the cluster have direct access to VM disk images. There is no need to copyVM image data, so live migration is very fast in that case.The storage library (package libpve-storage-perl) uses a flexible plugin system to provide a com-mon interface to all storage types. This can be easily adopted to include further storage types in future.8.1 Storage TypesThere are basically two different classes of storage types:Block level storageAllows to store large raw images. It is usually not possible to store other files (ISO, backups, ..) onsuch storage types. Most modern block level storage implementations support snapshots and clones.RADOS, Sheepdog and GlusterFS are distributed systems, replicating storage data to different nodes.File level storageThey allow access to a full featured (POSIX) file system. They are more flexible, and allows you to storeany content type. ZFS is probably the most advanced system, and it has full support for snapshotsand clones.Table 8.1: Available storage typesDescription PVE type Level Shared Snapshots StableZFS (local) zfspool file no yes yesDirectory dir file no no1yesNFS nfs file yes no1yesCIFS cifs file yes no1yesGlusterFS glusterfs file yes no1yes Proxmox VE Administration Guide 84 / 328Table 8.1: (continued)Description PVE type Level Shared Snapshots StableLVM lvm block no2no yesLVM-thin lvmthin block no yes yesiSCSI/kernel iscsi block yes no yesiSCSI/libiscsi iscsidirectblock yes no yesCeph/RBD rbd block yes yes yesSheepdog sheepdog block yes yes betaZFS overiSCSIzfs block yes yes yes1: On file based storages, snapshots are possible with the qcow2 format.2: It is possible to use LVM on top of an iSCSI storage. That way you get a shared LVM storage.8.1.1 Thin ProvisioningA number of storages, and the Qemu image format qcow2, support thin provisioning. With thin provisioningactivated, only the blocks that the guest system actually use will be written to the storage.Say for instance you create a VM with a 32GB hard disk, and after installing the guest system OS, the rootfile system of the VM contains 3 GB of data. In that case only 3GB are written to the storage, even if theguest VM sees a 32GB hard drive. In this way thin provisioning allows you to create disk images which arelarger than the currently available storage blocks. You can create large disk images for your VMs, and whenthe need arises, add more disks to your storage without resizing the VMs’ file systems.All storage types which have the “Snapshots” feature also support thin provisioning.CautionIf a storage runs full, all guests using volumes on that storage receive IO errors. This can cause filesystem inconsistencies and may corrupt your data. So it is advisable to avoid over-provisioning ofyour storage resources, or carefully observe free space to avoid such conditions.8.2 Storage ConfigurationAll Proxmox VE related storage configuration is stored within a single text file at /etc/pve/storage.cfg. As this file is within /etc/pve/, it gets automatically distributed to all cluster nodes. So all nodesshare the same storage configuration.Sharing storage configuration make perfect sense for shared storage, because the same “shared” storageis accessible from all nodes. But is also useful for local storage types. In this case such local storage isavailable on all nodes, but it is physically different and can have totally different content. Proxmox VE Administration Guide 85 / 3288.2.1 Storage PoolsEach storage pool has a <type>, and is uniquely identified by its <STORAGE_ID>. A pool configurationlooks like this:<type>: <STORAGE_ID><property> <value><property> <value>...The <type>: <STORAGE_ID> line starts the pool definition, which is then followed by a list of proper-ties. Most properties have values, but some of them come with reasonable default. In that case you can omitthe value.To be more specific, take a look at the default storage configuration after installation. It contains one speciallocal storage pool named local, which refers to the directory /var/lib/vz and is always available. TheProxmox VE installer creates additional storage entries depending on the storage type chosen at installationtime.Default storage configuration (/etc/pve/storage.cfg)dir: localpath /var/lib/vzcontent iso,vztmpl,backup# default image store on LVM based installationlvmthin: local-lvmthinpool datavgname pvecontent rootdir,images# default image store on ZFS based installationzfspool: local-zfspool rpool/datasparsecontent images,rootdir8.2.2 Common Storage PropertiesA few storage properties are common among different storage types.nodesList of cluster node names where this storage is usable/accessible. One can use this property torestrict storage access to a limited set of nodes.contentA storage can support several content types, for example virtual disk images, cdrom iso images,container templates or container root directories. Not all storage types support all content types. Onecan set this property to select for what this storage is used for. Proxmox VE Administration Guide 86 / 328imagesKVM-Qemu VM images.rootdirAllow to store container data.vztmplContainer templates.backupBackup files (vzdump).isoISO imagessharedMark storage as shared.disableYou can use this flag to disable the storage completely.maxfilesMaximum number of backup files per VM. Use 0 for unlimited.formatDefault image format (raw|qcow2|vmdk)WarningIt is not advisable to use the same storage pool on different Proxmox VE clusters. Some stor-age operation need exclusive access to the storage, so proper locking is required. While this isimplemented within a cluster, it does not work between different clusters.8.3 VolumesWe use a special notation to address storage data. When you allocate data from a storage pool, it re-turns such a volume identifier. A volume is identified by the <STORAGE_ID>, followed by a storage typedependent volume name, separated by colon. A valid <VOLUME_ID> looks like:local:230/example-image.rawlocal:iso/debian-501-amd64-netinst.isolocal:vztmpl/debian-5.0-joomla_1.5.9-1_i386.tar.gziscsi-storage:0.0.2.scsi-14 ←-f504e46494c4500494b5042546d2d646744372d31616d61To get the file system path for a <VOLUME_ID> use:pvesm path <VOLUME_ID>