Week 3: Wired up

Three files, one lookup

When a program asks for db01.kibbleco.example, the C library first reads /etc/nsswitch.conf to learn the order of sources. The hosts: line usually says files dns: try /etc/hosts (static name-to-address lines) first, then DNS using the servers in /etc/resolv.conf. The same file also points users and groups at files, LDAP or SSSD.

# /etc/nsswitch.conf
hosts:  files dns
passwd: files sss

# /etc/hosts
127.0.0.1    localhost
10.20.30.15  db01.kibbleco.example db01

# /etc/resolv.conf
search kibbleco.example
nameserver 10.10.0.53

Exam tip A wrong line in /etc/hosts beats correct DNS, because files come first. Check it when one server sees an old address.

Who writes resolv.conf?

On modern systems you rarely edit /etc/resolv.conf by hand. On Ubuntu it's a link to the systemd-resolved stub (nameserver 127.0.0.53), and NetworkManager rewrites it on RHEL. Set DNS servers in the network configuration (Netplan or the NetworkManager profile) and let the tools write the file. search lets you type db01 instead of the full name.

Dev: If your fix disappears overnight, something else owns that file.

Exam tip Hand edits to a managed resolv.conf vanish at the next reboot or DHCP renewal.

NetworkManager with nmcli

NetworkManager runs the network on RHEL, Fedora and most desktops. Settings live in connection profiles; devices are the actual NICs. nmcli changes profiles from the shell, and nmtui is a text menu for the same thing. Changes to a profile apply when you bring the connection up again.

$ nmcli device status
$ nmcli connection show
$ sudo nmcli con mod ens1f0 \
   ipv4.method manual \
   ipv4.addresses 10.20.30.15/24 \
   ipv4.gateway 10.20.30.1 \
   ipv4.dns 10.10.0.53
$ sudo nmcli con up ens1f0

Exam tip con mod saves, con up activates. The objectives print "nmconnect": read it as nmcli connection and nmtui.

Netplan on Ubuntu

Ubuntu servers describe the network in YAML files in /etc/netplan/. Netplan hands them to a renderer (systemd-networkd on servers, NetworkManager on desktops). YAML is picky: indent with spaces, never tabs.

# /etc/netplan/50-kibble.yaml
network:
  version: 2
  ethernets:
    ens3:
      addresses: [10.10.5.20/24]
      routes:
        - to: default
          via: 10.10.5.1
      nameservers:
        addresses: [10.10.0.53]

try, apply, status

netplan try applies the new config and rolls it back automatically after 120 seconds unless you press Enter to confirm. That saves you when a typo cuts off your own SSH session. netplan apply applies it for real, with no safety net. netplan status shows the current state of interfaces, addresses, routes and DNS.

Exam tip Working remotely? Use netplan try. A wrong apply over SSH can lock you out until someone visits the console.

The ip command

ip replaced the old ifconfig and route. ip address (ip a) shows addresses, ip link shows interfaces and whether they are UP, and ip route shows the routing table, including the default gateway. hostname prints or sets the name (hostnamectl makes it permanent). arp (or ip neigh) shows the MAC addresses this host has learned on the local network.

$ ip -br a
lo      UNKNOWN 127.0.0.1/8
ens3    UP      10.10.5.20/24
$ ip route
default via 10.10.5.1 dev ens3
10.10.5.0/24 dev ens3 proto kernel

Exam tip ip changes made on the command line are lost at reboot. Make them permanent in Netplan or NetworkManager.

Is it reachable, and where does it stop?

ping / ping6 test basic reachability over ICMP (IPv4 / IPv6). traceroute lists each router on the path. tracepath does the same without root and also reports the path MTU. mtr combines ping and traceroute live, showing loss and delay at every hop, which is great for intermittent problems.

Exam tip A server that ignores ping can still be up: many firewalls drop ICMP. Test the real service port too.

Names, ports and services

dig and nslookup query DNS directly (dig +short db01.kibbleco.example; dig @10.10.0.53 to ask one server). curl talks HTTP(S): curl -I shows headers, -v shows the TLS and HTTP conversation. nc (netcat) tests a TCP port (nc -zv db01 5432) or makes a quick listener. ss -tulpn lists listening sockets and the process behind each. nmap scans another host to see which ports answer from the outside.

Exam tip ss = what this host listens on. nmap = what another host exposes. nc = can I reach this one port?

Looking at the wire and the link

tcpdump captures packets: tcpdump -i ens3 port 53 shows DNS traffic live, and -w saves a file for Wireshark. ethtool ens3 shows link speed, duplex, auto-negotiation and whether a link is detected. iperf3 measures throughput between two hosts: run iperf3 -s on one and iperf3 -c <server> on the other.

Exam tip Slow transfers? iperf3 measures the network alone, which separates network trouble from disk or app trouble.

Link, address, route, name

Work from the bottom. Link down: ip link says NO-CARRIER or state DOWN: cable, switch port, NIC. Link negotiation issues: ethtool shows 100Mb/s or Half duplex on a gigabit link: auto-negotiation mismatch, fix both ends. Subnet: a wrong prefix (/16 instead of /24) reaches some hosts and not others. Routing issues: no default route, or a wrong gateway, means the local net works and everything else says Network is unreachable.

Exam tip Order of checks: link, address/subnet, gateway, DNS, firewall, the service itself.

DHCP, DNS and IP conflicts

DHCP issues: no address, or a leftover address, after boot. Check the NetworkManager or networkd logs (journalctl) for lease errors and whether the DHCP server's scope is full. DNS issues: the IP works but the name fails: check resolv.conf, then dig @server name. IP conflicts: two hosts claim one address, so connections flap. Neighbors' ARP caches show the MAC changing, and IPv6 logs "duplicate address detected".

Exam tip Ping by IP works but by name fails? That's DNS, every time.

MTU, bonding and MAC addresses

MTU mismatch: small packets pass, big transfers hang. Test with ping -M do -s 1472 (1472 + 28 header bytes = 1500) and lower the size until it passes. Bonding joins NICs for failover or speed. An LACP (802.3ad) bond needs a matching switch config; check /proc/net/bonding/bond0. MAC spoofing: a changed or cloned MAC can trip switch port security, or break DHCP reservations tied to the old MAC.

Exam tip SSH logs in fine but hangs at the first big output? Suspect MTU.

Firewalls, dual stack and "unreachable"

Misconfigured firewalls: ss shows the service listening, but clients time out. Test with nc from a client, then read the rules (next month you'll learn firewalld and ufw). Dual stack issues: a name returns both an AAAA and an A record, the client tries IPv6 first, and the IPv6 path is broken, so connections stall. Compare curl -4 with curl -6. Cannot ping server / server unreachable: ICMP may be blocked on purpose, so test the actual service port before declaring it down.

Exam tip curl -4 works and curl -6 hangs? Fix the IPv6 route or DNS, not the app.

Three files

Name resolution on Linux is controlled by three files:

• /etc/hosts: static name-to-IP entries (for example 10.0.0.5 db01). Checked before DNS by default
• /etc/resolv.conf: DNS servers (nameserver) and search domains. Often managed by systemd-resolved or NetworkManager, so edit their config instead
• /etc/nsswitch.conf: the order of sources, e.g. hosts: files dns means /etc/hosts first, then DNS

Troubleshooting

A name resolves to the wrong IP on one server only: look in /etc/hosts. Names don't resolve at all: check resolv.conf and test with dig name or nslookup. getent hosts name follows nsswitch.conf exactly like applications do.

What NetworkManager does

NetworkManager manages network connections on most RHEL, Fedora and desktop Ubuntu systems. Each connection is a saved profile (IP settings, DNS, Wi-Fi, bonds, VLANs) applied to a device.

Exam tip The objectives print 'nmconnect'; the standard tools are nmcli (command line) and nmtui (text menus). Learn nmcli connection commands, which is most likely what's meant.

nmcli essentials

The commands you'll use most:

• nmcli device status: devices and their state
• nmcli connection show: saved connections
• nmcli connection add type ethernet ifname eth0 con-name office ipv4.method manual ipv4.addresses 192.168.1.10/24 ipv4.gateway 192.168.1.1 ipv4.dns 1.1.1.1
• nmcli connection modify office ipv4.dns 8.8.8.8
• nmcli connection up office (apply changes) / down
• Profiles are stored in /etc/NetworkManager/system-connections/

Exam tip After nmcli connection modify, run nmcli connection up for the change to take effect.

Common faults

The faults that most often break a network interface:

• Wrong subnet/mask or gateway: can reach some hosts, not others. Check ip addr, ip route
• MTU mismatch: small packets work, large transfers hang (SSH connects but scp stalls). Test with ping -M do -s 1472 host; set MTU consistently (9000 jumbo frames only if the whole path supports them)
• Bonding: wrong mode or a bad member link; check /proc/net/bonding/bond0
• MAC spoofing / duplicate MAC or IP: intermittent connectivity; check ip neigh and switch logs
• Cannot ping server: link up (ip link, ethtool)? IP correct? Firewall dropping ICMP? Route back?

Work up the layers

Troubleshoot from the bottom of the network stack up:

• Layer 1-2: cable/link: ip link, ethtool eth0
• Layer 3: address and route: ip addr, ip route, ping the gateway, then the server
• Path: traceroute / tracepath / mtr to see where it stops
• Name: does DNS resolve? dig server
• Layer 4: is the service listening? ss -tulpn on the server; test from the client with nc -zv server 443
• Firewalls on both ends and in between

Exam tip Ping works but the app doesn't: the service, its port or a firewall. Name fails but IP works: DNS.