Month 2: People and processes

The three account files

/etc/passwd lists accounts, one per line, and everyone can read it: name, x (the password lives elsewhere), UID, GID, comment, home, shell. /etc/shadow holds the password hashes and aging dates, readable only by root. /etc/group lists groups and their extra members. getent passwd sam reads accounts through nsswitch.conf, so it also finds LDAP or SSSD users that aren't in the local files.

sam:x:1001:1001:Sam K:/home/sam:/bin/bash
www:x:986:986::/srv/www:/sbin/nologin

# /etc/group
warehouse:x:1050:rosa,tomas

Exam tip An x in field 2 of /etc/passwd just means "see /etc/shadow".

UIDs, GIDs and the effective IDs

The kernel tracks numbers, not names. UID identifies the user and GID the primary group. Each process also has an EUID and EGID (effective IDs): who it acts as right now. Usually they match; with sudo or a setuid program like passwd, the EUID becomes 0 (root) while the real UID stays yours. id shows uid, gid and groups.

$ id tomas
uid=1052(tomas) gid=1052(tomas)
  groups=1052(tomas),1050(warehouse)

User, system and service accounts

UID ranges tell you the kind of account (defaults in /etc/login.defs).

• root: UID 0, always
• System accounts: UID 1-999, made with useradd -r: no aging, usually no home
• Service accounts: system accounts a daemon runs as (nginx, postgres), with a nologin shell so nobody logs in as them
• User accounts: people, from UID 1000 up (UID_MIN in login.defs)

Exam tip Running each service as its own unprivileged account limits the damage if that service is broken into.

Add and delete

useradd is the low-level tool on every distro: useradd -m (make home) -s /bin/bash -G warehouse tomas. adduser is Debian/Ubuntu's friendly interactive script (on RHEL it's just a link to useradd). groupadd creates groups. To remove: userdel -r (home and mail too), Debian's deluser --remove-home, and groupdel. New homes are copied from /etc/skel; /etc/profile (plus /etc/profile.d/) sets the login environment for everyone.

Exam tip Want every new user to get a starter .bashrc or a README? Put it in /etc/skel.

Modify: the -aG trap

usermod changes an existing account: -s shell, -d home, -l login name, -e expiry date, and -G groups. usermod -aG docker sam ADDS a group. Without -a, -G REPLACES all of Sam's extra groups with just that one. chsh -s /bin/zsh changes a shell; passwd sets a password; groupmod -n renames a group.

Sam: I learned -aG the hard way. I removed myself from sudo once.

Ranger: sympathetic whine

Exam tip Classic trap: usermod -G wheel sam silently drops Sam from every other group. Always -aG.

Lock and expire

passwd -l or usermod -L locks the password by putting ! in front of the hash; -u / -U unlocks. That doesn't stop SSH key logins. To disable the whole account, expire it: chage -E 0 tomas (or usermod -e 1). chage -E 2027-01-15 sets an expiry date; chage -l lists the aging; -M sets max days between changes, -d 0 forces a password change at next login. Defaults for new accounts sit in /etc/login.defs and /etc/default/useradd.

$ sudo chage -l tomas
Password expires        : never
Account expires         : Jan 15, 2027

Who is here, and who was

whoami prints your user name; id your IDs and groups; groups just the groups. who lists logged-in users; w adds what they're running and the load average. last shows the login history (from /var/log/wtmp) including reboots; lastlog shows each account's most recent login, handy for spotting unused accounts.

Exam tip last = history of sessions. lastlog = one line per account, most recent login.

PIDs, parents and the tree

Every process has a PID and a parent, its PPID: the process that started it. systemd is PID 1, the ancestor of everything. pstree -p draws the family tree; ps -ef or ps aux lists every process (ps -eo pid,ppid,stat,cmd picks columns). Each one also has a directory /proc/<PID>/ with its command line, status, open file descriptors (fd/) and environment.

$ ps -eo pid,ppid,stat,cmd | head -4
  PID  PPID STAT CMD
    1     0 Ss   /sbin/init
  812     1 Ss   nginx: master
  813   812 S    nginx: worker

Live monitors

top shows CPU, memory and the busiest processes live (press P for CPU, M for memory, k to kill). htop is a friendlier, colored top with scrolling and tree view. atop also records history to disk, so you can replay what happened at 3 a.m. mpstat -P ALL shows each CPU core and pidstat shows per-process CPU, memory (-r) or disk (-d) over time; both come from the sysstat package.

Exam tip Want to know what was slow last night? atop's history or sysstat's sar files, not top.

Looking inside one process

lsof -p PID lists every file and socket the process has open. strace -p PID shows its system calls live: a process stuck on read() of a network socket, or failing open() on a missing file, tells you exactly what it's waiting for. Both are the first tools for "it's running but doing nothing".

Exam tip strace output full of ENOENT? It's looking for a file that isn't there.

Process states

The STAT column tells you what a process is doing.

• R running (or ready to run on a CPU)
• S sleeping: waiting for something, can be woken (most processes, most of the time)
• D blocked: uninterruptible sleep, usually waiting on disk or NFS I/O. Even kill -9 waits
• T stopped: paused by Ctrl+Z or SIGSTOP
• Z zombie: finished, but its parent hasn't collected the exit status yet

Exam tip You can't kill a zombie; it's already dead. Fix or restart its parent, which then reaps it.

nice and renice

The nice value runs from -20 (greediest, highest priority) to 19 (most polite); the default is 0. Start a job politely with nice -n 10 ./export.sh. Change a running one with renice -n 15 -p 3991. Only root may lower the value (raise priority); ordinary users can only be nicer. Process limits cap what one user or service can use: ulimit -a shows them, /etc/security/limits.conf sets them per user, and LimitNOFILE= does it in a systemd unit.

Exam tip Higher nice number = LOWER priority. Nice people let others go first.

Signals and kill

kill sends a signal, by default 15 SIGTERM: please clean up and exit. 9 SIGKILL can't be caught: the kernel ends the process at once, with no cleanup. 1 SIGHUP originally meant the terminal hung up; many daemons reload their config on it. killall nginx signals by exact name; pkill -u tomas by pattern or user.

kill 3991          # SIGTERM (15)
kill -HUP 812      # reload config
kill -9 3991       # last resort
pkill -u tomas     # all of his

Exam tip TERM first, wait, then KILL. kill -9 first can corrupt files the program was writing.

Jobs in your shell

& starts a command in the background. Ctrl+Z suspends the foreground job (state T); bg resumes it in the background, fg brings it back, jobs lists them. Ctrl+C sends SIGINT to stop it. Ctrl+D isn't a signal: it's end-of-input, which also logs you out of a shell. nohup cmd & ignores the HUP sent at logout, so a long job survives. exec cmd replaces the shell itself with cmd.

Exam tip Ctrl+Z pauses, it doesn't end. A forgotten stopped job still holds its files open.

Three schedulers, three jobs

crontab -e edits your recurring jobs (five time fields, then the command); crontab -l lists them. at runs something once: at 22:00, then type the commands; atq lists and atrm removes queued jobs. anacron is for machines that aren't always on: it runs daily, weekly and monthly jobs that were missed while the machine was off. Its table, /etc/anacrontab, counts the period in days.

# min hour dom mon dow command
  30  2    *   *   1-5 /opt/sync.sh

$ echo /usr/local/bin/check | at 23:30

Exam tip Laptop asleep at 3 a.m.? cron skips that run; anacron catches up when it wakes.

Killed, unresponsive, leaking, crashing

Four classic process problems:

• Killed processes: the OOM killer ended them when RAM ran out. dmesg or journalctl -k shows "Out of memory: Killed process"
• Unresponsive process: check its state. In D it waits on I/O (fix the disk or NFS); otherwise strace it, then TERM, then KILL
• Memory leaks: memory use (RSS) climbs for days and never drops. Track it with ps or pidstat -r; restart as a stopgap, report the bug
• Segmentation fault: the program touched memory it doesn't own: a bug, a bad library or failing RAM. coredumpctl lists crash dumps

Exam tip A segfault after an update often means mismatched libraries. Reinstall the package before blaming RAM.

The commands

Creating accounts and groups:

• useradd -m -s /bin/bash -G wheel alice: create a user with a home directory (-m), shell, and supplementary group
• adduser alice: Debian/Ubuntu's friendly script that asks questions and creates the home; on RHEL it's just a link to useradd
• groupadd developers: create a group (-g to choose the GID)
• passwd alice: set the password (useradd doesn't)

Where defaults come from

useradd reads /etc/default/useradd and /etc/login.defs (UID ranges, password aging) and copies /etc/skel into the new home.

The commands

Removing accounts and groups:

• userdel alice: remove the account but keep the home directory
• userdel -r alice: also remove the home directory and mail spool
• deluser --remove-home alice: Debian/Ubuntu equivalent
• groupdel developers: remove a group (fails if it's someone's primary group)

Before deleting

Check for running processes (pkill -u alice), cron jobs, and files elsewhere: find / -user alice. Files left behind show a number instead of a name (orphaned UID).

Exam tip Staff member left but you must keep their files for review: lock the account first, delete later.

The commands

Changing existing accounts and groups:

• usermod -aG docker alice: add to a supplementary group (-a is essential, or other groups are replaced)
• usermod -s /bin/zsh alice or chsh -s /bin/zsh alice: change shell
• usermod -l newname oldname, -d /new/home -m: rename, move home
• groupmod -n newname oldname: rename a group
• passwd alice: change password

Check the result

id alice shows UID, GID and groups. Group changes apply at the user's next login (or with newgrp).

Exam tip usermod -G docker alice without -a removes alice from all her other supplementary groups.

Three ways

You can block an account in different ways, with different effects:

• passwd -l alice / usermod -L alice: lock the password (puts ! in front of the hash in /etc/shadow); unlock with passwd -u or usermod -U
• chage -E 0 alice (or usermod -e 1970-01-02): expire the account so no login works at all
• Set the shell to /sbin/nologin

Important difference

Locking the password doesn't stop SSH key logins. To fully block someone, expire the account (chage -E 0) or set the shell to nologin too. passwd -S alice shows the status (L = locked).

chage

chage -l alice lists the settings. Change them with:

• -M 90: maximum days between password changes
• -m 1: minimum days before changing again
• -W 7: warning days
• -I 14: inactive days after expiry before the account locks
• -E 2026-12-31: account expiration date
• -d 0: force a password change at next login

Defaults for new users

/etc/login.defs sets defaults (PASS_MAX_DAYS, PASS_MIN_DAYS, PASS_WARN_AGE); they apply to accounts created afterwards. The values are stored per user in /etc/shadow.

Two templates

/etc/skel holds files copied into each new home directory (.bashrc, .bash_profile, default folders). Changes don't affect existing users. /etc/profile (and **/etc/profile.d/*.sh) runs for every** login shell, for all users, existing and new.

Exam tip Give every future user a standard .bashrc: /etc/skel. Set an environment variable for everyone now: a script in /etc/profile.d/.

/etc/passwd

Readable by everyone. One line per account: name:x:UID:GID:comment:home:shell, for example alice:x:1001:1001:Alice:/home/alice:/bin/bash. The x means the password hash is in shadow.

/etc/shadow and /etc/group

/etc/shadow (root only) holds the password hash and aging fields: name:hash:lastchange:min:max:warn:inactive:expire. A hash starting with ! or * means locked/no password. /etc/group lists groups: name:x:GID:members (supplementary members). /etc/gshadow holds group passwords.

Exam tip Edit them safely with vipw / vigr, or better, with useradd/usermod/chage.

Real vs effective IDs

Each process carries both real and effective identities:

• UID: the user's numeric ID (root = 0)
• GID: the primary group's numeric ID
• EUID / EGID: the effective IDs used for permission checks while a process runs. Usually the same as the real IDs, but a setuid program runs with the EUID of its owner (passwd runs as root so it can write /etc/shadow)

See them

id shows uid, gid and groups. ps -o user,euser,ruser -p PID compares effective and real users of a process.

UID ranges

The UID tells you what kind of account it is:

• 0: root
• 1-999 (older systems 1-499): system and service accounts for daemons (apache, mysql, sshd). No interactive login: shell /sbin/nologin, often no home
• 1000+: regular user accounts (set by UID_MIN in /etc/login.defs)

Creating one

useradd -r -s /sbin/nologin myservice creates a system account in the system range. Running each service under its own account limits the damage if it's compromised.

Parents and children

Every process has a PID (process ID) and a PPID (the PID of the parent that started it). systemd is PID 1, the ancestor of everything. ps -ef or ps -eo pid,ppid,cmd shows both; pstree -p draws the tree.

Exam tip Killing a parent doesn't always kill its children; orphans are adopted by PID 1.

The STAT column

In ps aux or top, the state letter tells you what a process is doing:

• R running or runnable
• S sleeping (waiting for an event; most processes)
• D uninterruptible sleep, blocked on I/O (often disk or NFS); can't be killed until I/O returns
• T stopped (Ctrl+Z or SIGSTOP); resume with fg/bg or SIGCONT
• Z zombie: finished, but the parent hasn't collected its exit status

Zombies

A zombie uses no CPU or memory, only a process table entry. You can't kill a zombie; fix or restart its parent (find it with the PPID) so it reaps the child.

Exam tip Many processes in D state + high load average = I/O problem, not CPU.

Niceness

The nice value runs from -20 (highest priority) to 19 (lowest); default 0. A 'nicer' process gives CPU time to others.

• nice -n 10 ./backup.sh: start with lower priority
• renice -n 5 -p 1234: change a running process
• Only root can lower the value (raise priority) below the current value
• top shows NI; press r in top to renice

cron and crontab

crontab -e edits your schedule; crontab -l lists it. Format: minute hour day-of-month month day-of-week command, for example **30 2 1-5 /opt/backup.sh (2:30 weekdays). System jobs go in /etc/crontab (with a user field) and /etc/cron.d/; /etc/cron.daily** etc. hold scripts.

anacron and at

anacron runs daily/weekly/monthly jobs that were missed because the machine was off (laptops, desktops); configured in /etc/anacrontab. at runs a command once at a time: at 22:00, then type commands, Ctrl+D; atq lists, atrm removes.

Exam tip Recurring on a server that's always on: cron. Recurring on a machine that may be off: anacron. Once only: at. (systemd timers are the modern alternative.)