Linux with Operating System Concepts

Name
Processor 
Info
Process 
Info
Memory 
Info
VM 
Info
File 
System 
Info
I/O 
Info
Network 
Info
Comments
df
*
du
*
free
*
*
iostat
*
*
*
*
lpstat
*
mpstat
*
netstat
*
Obsolete, replaced by ss
nmap
*
nstat
*
pidstat
*
*
*
ps
*
*
*
*
*
rtacct
*
System monitor
*
*
*
*
*
*
*
Graphical, persistent
sar
*
*
*
*
*
*
*
ss
*
stat
*
strace
*
*
top
*
*
*
*
Persistent
uptime
*
System uptime
vmstat
*
*
*
*
who
*
Lists logged-in users

---

596
◾
Linux with Operating System Concepts
You might also find critical messages are being sent to *, meaning every open console. 
Let us step through these files to explore what you can expect to find in each (we omit mail-
log and spooler).
Log files tend to store entries that have the following information:
• Date and time of the event
• Host name of the computer on which the event arose
• Name of the program that generated the log message
• A (short) description of the event
The message will generally be short (less than one line). If the process that generated the 
message is not one of syslogd (rsyslogd) or the kernel, the PID is also included.
As the 
messages
file contains entries generated from both syslogd (rsyslogd) and 
the kernel via klogd, you might find the entries here overlap the information obtained 
by 
dmesg
. Recall from Chapter 11 that dmesg responds with the kernel ring, a complete 
listing of the events during the last kernel initialization. Any syslogd (rsyslogd) generated 
events should be listed in the messages file after the kernel initialization messages. The sys-
logd generated events are provided by various email (mail), authentication (authpriv), and 
scheduling (cron) programs whose events are denoted with a priority of none. You might 
notice that all other events generated by these three types of programs are sent to other files 
(/var/log/maillog, /var/log/secure, and /var/log/cron). Also sent to messages are all events 
with a priority of info.
The secure log stores information based on authorization and authentication events. 
These events occur when users attempt to log in to the system or must otherwise provide 
a password (e.g., when using su or sudo). The secure file will contain entries that look like 
the following:
• 
Nov 23 10:29:16 mycomputer sshd[1781]: Server listening on 0.0.0.0 
port 22.
• 
Nov 23 10:29:41 mycomputer pam: gdm-password[2041]: pam _ unix(gdm-
password:session): session opened for user foxr by (uid = 0)
• 
Nov 23 10:32:18 mycomputer su: pam _ unix(su:session): session 
opened for user root by foxr(uid = 500)
These three messages have taken place within a few minutes of each other. The first 
message indicates that someone is attempting to log in using ssh over port 22. The next 
message indicates that a log-in attempt was made through the PAM authentication mod-
ule by user foxr. The entry 
uid 
=
0
means that root handled the log-in attempt. The next 
line indicates that a user attempted to use su. In this case, the user was foxr (
uid 
=
500
) 
attempting to log in as root. Note that 
mycomputer
is the hostname of the computer that 
generated these messages.

---

Maintaining and Troubleshooting Linux
◾
597
The following entries show messages in the 
secure
log file that arose because of a 
failed log in to root via su.
• 
Nov 23 11:37:20 mycomputer su: pam_unix(su:session): session 
opened for user root by foxr(uid = 500)
• 
Nov 23 11:37:27 mycomputer unix_chkpwd[4993]: password check 
failed for user (root)
• 
Nov 23 11:37:27 mycomputer su: pam_unix(su:auth): authentication 
failure; logname = foxr uid = 500 euid = 0 tty = ptrs/1 ruser = foxr 
rhost = user = root
Here, we see a logged event of an su attempt followed by two messages indicating failed 
authentication, one from 
unix_chkpwd
and one from 
su
. Logged with the events are 
the user’s username who attempted to but failed to log in and the username that this user 
attempted to su to.
The log file /
var/log/cron
contains entries generated from crond, anacron, and run-
parts. These log entries, like the previous examples contain the date/time, host, program, 
and event. In the case that the event was executed by crond, then the user who issued the 
crontab job is also recorded. For instance, you might find entries like the following:
• 
Nov 20 11:01:01 mycomputer anacron[5013]: Anacron started on 
2012-11-20
• 
Nov 20 15:10:01 mycomputer CROND[5042]: (foxr) CMD (./my_
scheduled_script >> output.txt)
• 
Nov 20 16:43:01 mycomputer CROND[5311]: (foxr) CMD (echo “did 
this work?”)
• 
Nov 20 16:44:01 mycomputer CROND[5314]: (foxr) CMD (echo “did 
this work?”)
The first entry informs us when anacron is starting. The other three entries are of 
crontab jobs submitted by user foxr. The first submission executes a script while the other 
two occur one minute apart and merely perform an echo statement.
Next, we see two logged events from run-parts that are running programs found in /etc/
cron.daily. For any events run by run-parts, there should be two entries, one that indicates 
when the event started and one of when it finished. The two entries are of the same event, 
logrotate.
• 
Nov 18 03:49:01 mycomputer
run-parts(/etc/cron.daily)[14361]: starting logrotate
• 
Nov 18 03:49:01 mycomputer
run-parts(/etc/cron.daily)[14364]: finishing logrotate

---

598
◾
Linux with Operating System Concepts
The 
boot.log
file contains entries of logged events during the previous boot attempt, 
specifically initialization steps. These entries, unlike the previous log file entries, will be 
segmented into groups based on the phase of the initialization step. Rather than logging 
these by time, they are listings of the events and the results (success or failure). Below are 
some excerpts from this file, with comments interspersed.
Starting udev:
Setting hostname mycomputer:
[ OK ]
Message indicates hostname has been established.
Checking filesystems
/dev/sda1: clean, 93551/256000 files, 784149/1024000 blocks
/dev/sda5: recovering journal
/dev/sda5: clean, 472/40320 files, 26591/16180 blocks
[ 
OK 
]
Remounting root file system in read-write mode:
[ OK ]
Mounting local filesystems: 
[ OK ]
Enabling local file system quotas:
[ OK ]
Enabling /etc/fstab swaps: 
[ OK ]
The root file system along with the /home file system is checked by fsck. We see both 
file systems are clean (no corruption found). This is followed by mounting the file systems. 
First, the root file system is remounted in read–write mode as earlier it was read-only mode. 
All file systems mounted as per the /etc/fstab file. File system disk quotas enabled (if any).
Iptables: Applying firewall rules:
[ OK ]
Bringing up loopback interface: 
[ OK ]
Bringing up interface eth0: 
[ OK ]
Network service started bringing up both available interface devices. Firewall started 
with no errors (errors could be caused by having syntactically improper rules).
Starting 
auditd: 
[ 
OK 
]
Starting 
portreserve: 
[ 
OK 
]
Starting system logger:
[ OK ]
Starting 
irqbalance: 
[ 
OK 
]
Starting 
crond: 
[ 
OK 
]
Starting 
atd: 
[ 
OK 
]
Starting 
certmonger: 
[ 
OK 
]
Other services started successfully.
14.5.2 Audit Logs
A log subdirectory of note is /
var/log/audit
. This subdirectory stores the audit dae-
mon log files. The audit daemon (
auditd
) is responsible for logging general activities 
of the users. Such activities will include logins, login attempts, failed logins, opening of 

---

Maintaining and Troubleshooting Linux
◾
599
terminal windows, launching of executable programs, system calls, failed system calls, 
cryptographic events, and cryptography keys generated among numerous other activities.
The audit log files are quite large; so, we will view their content using two handy pro-
grams. First, there is 
aureport
that summarizes the log entries, providing totals for the 
various event categories. Second is 
ausearch
that the system administrator will use to 
query the audit log files for specific types of event information.
For ausearch, you specify a type of event you are interested and in some cases, a specific 
value to match against. You might, for instance, request any log entry of a user given the 
UID or any log entry of a given process given the PID. Tables 14.5 and 14.6 provide options 
for aureport and ausearch, respectively. The ausearch program is more complex but pro-
vides specific audit entries rather than summaries from aureport.
Running aureport with no options gives the following output:
Summary Report
======================
Range of time in logs: 03/19/2013 10:11:02.774 –
04/16/2013 10:21:15.081
TABLE 14.6 
Ausearch Options