Linux Diagnostics

Linux Diagnostics Guide

Linux provides a deep set of command-line tools for inspecting system state, troubleshooting performance issues, checking hardware, reviewing logs, and validating security posture.

System Overview

System Information

Check the operating system, kernel, CPU architecture, and basic host details:

uname -a                            # Show kernel, hostname, architecture, and OS details
hostnamectl                         # Show hostname, OS, kernel, and machine information
lscpu                               # Show CPU architecture and processor details
cat /etc/os-release                 # Show Linux distribution and version

Uptime and Load

Review how long the system has been running and whether it is under load:

uptime                              # Show uptime, users, and load averages
w                                   # Show logged-in users and load averages
vmstat 1                            # Show CPU, memory, swap, and I/O stats every second
last reboot                         # Show reboot history

CPU and Memory

CPU Performance

Monitor process activity, CPU utilization, and CPU frequency information:

top                                 # Interactive process and CPU monitor
htop                                # Enhanced interactive process monitor
mpstat -P ALL 1                     # Show CPU usage per core every second
cpufreq-info                        # Show CPU frequency governor and frequency details

Install optional tools when needed:

sudo apt install htop sysstat cpufrequtils

Memory Diagnostics

Inspect memory usage, swap activity, and process-level memory consumption:

free -h                             # Show used and available memory
watch -n 1 free -h                  # Refresh memory summary every second
vmstat 1                            # Watch memory, swap, I/O, and CPU activity
smem                                # Show proportional memory usage by process

Install smem if it is not available:

sudo apt install smem

Disk and Filesystems

Filesystem Usage

Check mounted filesystems, directory sizes, partitions, and block devices:

df -h                               # Show filesystem disk usage
du -h --max-depth=1                 # Show directory sizes one level deep
lsblk                               # List disks, partitions, and mount points
findmnt                             # Show mounted filesystems as a tree

Filesystem Checks

Inspect and repair filesystems carefully. Run fsck on unmounted filesystems whenever possible:

sudo fsck /dev/sdX                  # Check and repair filesystem errors
sudo badblocks -v /dev/sdX          # Scan a device for bad blocks

Storage Performance and Health

Measure disk performance and review SMART health data:

sudo hdparm -tT /dev/sdX            # Test buffered and cached disk reads
sudo smartctl -a /dev/sdX           # Show SMART health and device diagnostics

Install SMART tooling if needed:

sudo apt install smartmontools

Processes and Services

Process Diagnostics

List processes, find specific services, and stop hung processes:

ps aux                              # List active processes
pgrep -a <process_name>             # Find matching processes with full command line
top                                 # Monitor active processes in real time
kill <pid>                          # Ask a process to terminate
kill -9 <pid>                       # Force-kill a hung process

Systemd Services

Inspect system services and troubleshoot service failures:

systemctl list-units --type=service # List loaded services
systemctl status <service_name>     # Show service status and recent logs
systemctl restart <service_name>    # Restart a service
journalctl -u <service_name> -xe    # Show detailed logs for a service

Network Diagnostics

Interfaces and Connectivity

Inspect network interfaces, routes, DNS, and basic connectivity:

ip a                                # Show network interfaces and addresses
ip route                            # Show routing table
resolvectl status                   # Show DNS resolver status
ping google.com                     # Test basic connectivity
traceroute google.com               # Trace network path to a host

Ports and Sockets

Find listening ports and active network connections:

ss -tuln                            # Show TCP and UDP listening sockets
ss -tunap                           # Show sockets with process details
netstat -tuln                       # Older alternative to ss
sudo lsof -i -P -n                  # Show processes using network sockets

Network Speed

Measure public internet speed when speedtest-cli is installed:

speedtest-cli                       # Run an internet speed test
sudo apt install speedtest-cli      # Install speedtest-cli on Debian-based systems

Logs and Kernel Diagnostics

System Logs

Use journalctl for systemd logs and files under /var/log for distribution-specific logs:

journalctl -xe                      # Show recent high-priority system log entries
journalctl -b                       # Show logs from the current boot
journalctl --since "1 hour ago"     # Show logs from a time range
cat /var/log/auth.log               # Show authentication logs on Debian-based systems

Kernel Messages and Modules

Inspect kernel messages and loaded modules:

dmesg | tail -n 50                  # Show recent kernel messages
lsmod                               # List loaded kernel modules
dmesg | grep <module_name>          # Search kernel messages for a module
sudo modprobe -r <module_name>      # Remove a kernel module

Hardware Information

Device Inventory

List hardware, PCI devices, USB devices, and power status:

sudo lshw                           # Show detailed hardware overview
lspci                               # List PCI devices
lsusb                               # List USB devices
upower -i /org/freedesktop/UPower/devices/battery_BAT0  # Show laptop battery details

Package Management

Debian-Based Systems

Inspect installed packages, search package repositories, and update the system:

dpkg --get-selections               # List installed package selections
apt search <package_name>           # Search available packages
sudo apt update                     # Refresh package metadata
sudo apt upgrade                    # Upgrade installed packages

Security Diagnostics

Local Exposure

Check local services, permissions, and privileged files:

sudo nmap -sT localhost             # Scan local TCP services
ss -tuln                            # Show listening sockets
ls -l                               # Show file permissions and ownership
find / -perm -4000 2>/dev/null      # Find SUID binaries

Backup and Recovery

Disk Image Backups

Create and compress a raw disk or partition image:

sudo dd if=/dev/sdX of=backup.img bs=64K conv=noerror,sync status=progress
gzip backup.img

Use dd with care because reversing if and of can overwrite the wrong device.