Month 2: Backups and virtual machines

tar: the archiver everyone uses

tar bundles files into one archive and keeps permissions and owners. The letters: c create, x extract, t list, f the archive file, v verbose, -C extract into a directory. Add a compressor: z gzip, j bzip2, J xz.

tar -czf etc.tar.gz /etc     # create
tar -tzf etc.tar.gz          # list
tar -xzf etc.tar.gz -C /tmp  # extract
tar -cJf logs.tar.xz /var/log/app

Exam tip f must be followed by the archive name. tar -cfz backup ... names the archive "z".

cpio: archives from a file list

cpio reads the list of files to pack from standard input, which pairs nicely with find. -o creates (copy-out), -i extracts (copy-in), -d creates directories while extracting. You meet it in the wild because initramfs images are cpio archives.

find /srv/shop -mtime -1 |
  cpio -o > today.cpio
cpio -id < today.cpio

Exam tip tar takes file names as arguments. cpio takes them on stdin.

Compression tools compared

Each tool trades speed for size.

• gzip (.gz): fast, everywhere; gunzip or gzip -d to undo
• bzip2 (.bz2): smaller files, slower
• xz (.xz): smallest of the three, slowest to compress, fast to decompress; -T0 uses all CPU cores
• 7-Zip (7z, .7z): high ratio, also reads many formats; handy for Windows partners
• unzip (.zip): opens zip files from Windows and Macs (zip makes them)

Exam tip Size, smallest to biggest, usually: xz < bzip2 < gzip. Speed: the reverse.

Reading compressed files in place

Rotated logs are often .gz. You don't need to unpack them: zcat prints a .gz file, zless pages through it, zgrep searches it. zgrep 'Failed password' /var/log/auth.log.*.gz searches every old log at once.

Exam tip z-tools are for gzip files. xz has xzcat/xzgrep; bzip2 has bzcat.

dd: copy raw blocks

dd copies bytes from one file or device to another, with no idea of files or filesystems: if= is the input, of= the output, bs= the block size. It images whole disks, writes ISO files to USB sticks, and wipes devices. It also never asks for confirmation.

dd if=/dev/sdb of=/backup/sdb.img \
   bs=4M status=progress
dd if=ubuntu.iso of=/dev/sdc \
   bs=4M conv=fsync

Exam tip Swap if= and of= by mistake and you overwrite the source. Read the line twice before Enter.

ddrescue: when the disk is dying

ddrescue is dd for failing disks. It copies the healthy areas first, skips bad spots, then goes back to retry them, and it keeps a map file so you can stop and resume. Copy the dying disk to an image, then repair or mount the image, never the sick disk.

ddrescue /dev/sdb /backup/sdb.img sdb.map

Exam tip A failing disk gets worse with every read. Image it first with ddrescue, then fsck the image.

rsync: copy only what changed

rsync compares source and destination and sends only the differences, locally or over SSH. -a (archive) keeps permissions, owners, times and links; -z compresses over the network; --delete removes files at the destination that are gone from the source (a true mirror); -n is a dry run. A trailing slash matters: src/ copies the contents of src, while src copies the directory itself.

rsync -az --delete /srv/shop/ \
  backup01:/backups/shop/

Sam: rsync for nightly copies, tar for snapshots in time, and test a restore every month.

Exam tip Try --delete with -n first. A wrong path plus --delete is a very fast way to lose a backup.

KVM and QEMU

KVM (Kernel-based Virtual Machine) is a kernel module that turns Linux into a hypervisor, using the CPU's virtualization features (Intel VT-x, AMD-V: look for vmx or svm in /proc/cpuinfo). QEMU emulates the rest of the machine (disks, NICs, firmware) and runs each VM as a process. Together they're the standard Linux virtualization stack. QEMU alone can also emulate other CPU types, slowly.

$ grep -cE 'vmx|svm' /proc/cpuinfo
64
$ lsmod | grep kvm
kvm_intel   487424  4
kvm        1425408  1 kvm_intel

Bare metal vs virtual machines

A bare-metal server runs its OS directly on the hardware: all of the performance, none of the flexibility. A virtual machine shares a host with others: you can snapshot, clone, resize and move it, at a small performance cost. Hypervisors that run directly on hardware (like KVM inside the Linux kernel) are type 1; ones that run as an app on a desktop OS are type 2.

Exam tip Need every bit of GPU or disk speed, or special hardware? Bare metal. Need flexibility and density? VMs.

Paravirtualized drivers: VirtIO

An emulated NIC (e1000) or disk (IDE, SATA) pretends to be real hardware, which costs CPU time. Paravirtualized drivers know they're in a VM and talk to the hypervisor directly. On KVM that's VirtIO: virtio-net for networking and virtio-blk or virtio-scsi for disks. Disks show up as /dev/vda instead of /dev/sda.

Exam tip Slow VM disk or network? Check whether it's on an emulated device instead of VirtIO.

Disk images and VM states

qemu-img handles disk images. info shows the image properties: format, virtual size, real size on disk, backing file. convert changes formats (raw, qcow2, vmdk), and resize grows an image (then grow the partition and filesystem inside the guest). VM states in virsh: running, paused (frozen in RAM), shut off, crashed, pmsuspended (guest went to sleep). Nested virtualization runs a hypervisor inside a VM: handy for labs and CI, slower in production.

qemu-img info web02.qcow2
qemu-img convert -f vmdk -O qcow2 \
   in.vmdk out.qcow2
qemu-img resize web02.qcow2 +20G

Exam tip qcow2 supports thin provisioning and snapshots; raw is simplest and fastest.

libvirt, virsh and virt-manager

libvirt is the management layer (the libvirtd or modular virtqemud daemons, plus an API) that tools talk to. virsh is its command line; virt-manager is the desktop GUI for the same thing.

virsh list --all          # every VM + state
virsh start web02
virsh shutdown web02      # polite (ACPI)
virsh destroy web02       # pull the plug
virsh suspend / resume web02
virsh autostart web02     # start at boot
virsh console web02

Exam tip destroy doesn't delete anything: it's a hard power-off. undefine removes the VM's definition.

VM operations

Everyday work on VMs:

• Resources: give a VM more vCPUs, RAM, disks or NICs (virsh setvcpus, setmem, attach-disk)
• Baseline image templates: a golden image, patched and generalized, that every new VM starts from
• Cloning: copy a VM (virt-clone). Then reset what must be unique: hostname, MAC, machine-id, SSH host keys (virt-sysprep does this)
• Snapshots: a point-in-time state to roll back to (virsh snapshot-create-as). Not a backup: same storage
• Migrations: move a VM to another host; live migration keeps it running, with shared storage

Exam tip A snapshot lives with the VM's disk. Lose the disk, lose the snapshots. Back up separately.

Virtual network types

How a VM reaches the world:

• NAT: libvirt's default network (virbr0, 192.168.122.0/24). VMs reach out; outside hosts can't reach in
• Bridged: the VM sits on the physical LAN like any server, with its own address there
• Host-only / isolated: VMs talk to each other and the host, nothing else
• Routed: the host routes the VM subnet without NAT; other routers need a route to it
• Open: like routed, but libvirt adds no firewall rules at all; you manage them

Exam tip Users must reach the VM as a normal server? Bridged. VM only needs updates? NAT is fine.

tar

tar bundles files into one archive (it doesn't compress unless you add a flag).

• tar -cvf backup.tar /etc: create
• tar -tvf backup.tar: list contents
• tar -xvf backup.tar -C /restore: extract into a directory
• Add -z (gzip, .tar.gz), -j (bzip2, .tar.bz2) or -J (xz, .tar.xz)

Exam tip c = create, x = extract, t = list, v = verbose, f = file name follows.

cpio

cpio reads a list of file names from standard input: find /etc | cpio -ov > etc.cpio creates; cpio -idv < etc.cpio extracts. It's used inside RPM packages and initramfs images.

The tools

Each tool trades speed for size differently:

• gzip / gunzip (.gz): fast, common
• bzip2 / bunzip2 (.bz2): smaller, slower
• xz / unxz (.xz): smallest of the three, slowest
• zip / unzip (.zip): compatible with Windows
• 7z (7-Zip, .7z): high compression, many formats (package p7zip or 7zip)

Remember

gzip, bzip2 and xz compress one file and replace it; combine with tar for directories. Read compressed logs without extracting: zcat, zless, zgrep (and xzcat, bzcat).

Exam tip Best compression ratio of the three classics: xz. Fastest: gzip.

How they fit together

KVM (Kernel-based Virtual Machine) is a Linux kernel module that turns Linux into a type 1 hypervisor using CPU virtualization extensions (Intel VT-x, AMD-V). QEMU emulates the hardware (disks, network cards) and runs the VMs, using KVM for speed. libvirt manages them (virsh, virt-manager, virt-install).

• Check CPU support: grep -E 'vmx|svm' /proc/cpuinfo
• lsmod | grep kvm shows the module loaded
• virsh list --all, virsh start vm1, virsh shutdown vm1

Why they're faster

Fully emulated hardware (like an emulated Intel network card) is slow because the hypervisor imitates real hardware. Paravirtualized drivers know they're in a VM and talk to the hypervisor directly. On KVM these are the VirtIO drivers: virtio-net (network), virtio-blk and virtio-scsi (disks), virtio-balloon (memory).

Exam tip Slow disk or network in a KVM guest: make sure it uses VirtIO devices. Linux has them built in; Windows guests need the virtio-win drivers.

Formats and commands

VM disks are files: qcow2 (QEMU's format: thin, snapshots, compression) or raw (simple, fast), plus VMDK and VHD(X) from other hypervisors.

• qemu-img info disk.qcow2: format, virtual size, actual size, backing file
• qemu-img convert -f vmdk -O qcow2 in.vmdk out.qcow2: convert formats
• qemu-img resize disk.qcow2 +20G: grow (then grow the partition and filesystem inside the guest)
• qemu-img create -f qcow2 disk.qcow2 40G: create

A VM inside a VM

Nested virtualization lets a guest act as a hypervisor itself, for labs, CI testing or running a hypervisor in the cloud. On KVM it's a module option: cat /sys/module/kvm_intel/parameters/nested (Y/1 = on); enable with options kvm_intel nested=1 (or kvm_amd) in /etc/modprobe.d/, and pass the host CPU features to the guest (cpu mode host-passthrough).

Exam tip Expect lower performance; it's for testing, not production.

Build once, deploy many

A template (golden image) is a prepared, patched VM image used to create new VMs quickly and consistently. Before making a template, generalize it: remove machine-specific data such as SSH host keys, machine-id, logs and user history (virt-sysprep does this), then use cloud-init to customize each new copy at first boot.

Exam tip Two cloned VMs with the same SSH host keys or machine-id: the template wasn't generalized.

Full vs linked clones

virt-clone --original vm1 --name vm2 --auto-clone copies a VM's configuration and disks. A full clone is an independent copy; a linked clone (backing file / qcow2 overlay) shares the original disk read-only and stores only changes, saving space but depending on the original.

Exam tip Clone a VM while it's shut down (or paused) so the disk is consistent, then change its hostname, IP and machine identity.

Live vs offline

Live migration moves a running VM to another host with almost no downtime: memory is copied while it runs, then a brief pause switches it over (virsh migrate --live vm1 qemu+ssh://host2/system). It usually needs shared storage (or storage migration) and compatible CPUs. Offline (cold) migration shuts the VM down and copies its disk and definition.

Exam tip Need to patch a host without stopping VMs: live-migrate them off first.

Compare

Where the operating system runs changes what you get:

• Bare metal: the OS runs directly on the hardware. Full performance and direct hardware access (GPUs, specialized cards); one OS per server; slower to provision
• Virtual machine: many isolated OSes share one physical host through a hypervisor. Better use of hardware, snapshots, templates, live migration; a little overhead
• Type 1 hypervisor runs on bare metal (KVM, ESXi, Hyper-V); type 2 runs on top of an OS (VirtualBox)

How a VM reaches the network

libvirt and other hypervisors offer several network modes:

• Bridged: the VM sits on the physical LAN like any other machine and gets an address from the real network
• NAT: the VM uses a private address and reaches out through the host (libvirt's default network, virbr0); outsiders can't connect in without port forwarding
• Host-only / isolated: VMs talk to each other and the host, not the outside world
• Routed: the host routes traffic for the VM subnet to the LAN without NAT (other routers must know the route)
• Open: libvirt creates the network but adds no firewall rules; you manage them

Choosing

Server VMs that other machines must reach: bridged. Quick outbound internet for a test VM: NAT. Malware lab: isolated.