My sysadmin tips

I will describe briefly some file undeletion technique using "Sleuth kit", a collection of UNIX-based command line tools that allow you to investigate a computer.

Note: For forensics analysis you should first create an image of the file system with 'dd' or dcfldd. A simple way is to boot a linux live CD (for example std) and to use dd to save the partition to an external USB disk. For example:
dd if=/dev/hda1 of=/mnt/usb_disk/image

Getting file ativity timeline
The file named 'image' is the image of the / partition previously saved with 'dd'
Lets use the fls and ils utilities from sleuthkit to gather informations about files and inodes activity,
fls -m / -r image > timeline
ils -m image >> timeline

And then, we can use the mactime utility to format data

# mactime -z CET -b image

Thu Jan 18 2007 11:50:07 
0 .a. -/-rw-r--r-- 0  0  12   /.titi.swp (deleted)
0 .a. -rw-r--r--   0  0  12   toto-dead-12
0 .a. -rw-r--r--   0  0  13   toto-dead-13
0 .a. -/-rw-r--r-- 0  0  13   /titi~ (deleted)
Thu Jan 18 2007 11:50:11 
0 m.c -/-rw-r--r-- 0  0  12   /.titi.swp (deleted)
0 m.c -/-rw-r--r-- 0  0  13   /titi~ (deleted)
0 m.c -rw-r--r--   0  0  13   toto-dead-13
19 mac -/-rw-r--r-- 0 0  14   /titi
0 m.c -rw-r--r--   0     12   toto-dead-12

Undeleting a data
You can use the fls utility to locate deleted files
fls -dar image
With the same file system image you can see:

r/r * 12: .titi.swp
r/r * 13: titi~

The letter r refer to files, d is for directory. The * mean deleted.

To recover data, we need to known the size of a block on the file system.

# fsstat image | grep "Block Size"
Block Size: 1024

If you now a string to search (for example a part of a log file...) you can use the Unix strings utility with -td arguments to known the offset of the string. In this example we are looking for the string "querty".

#strings -td image | grep "qwerty"

1265682 oqwertyuiop
1265694 qwertyuiop

Now we can calculus the number of bloc to jump

# echo 1265682/1024 | bc
1236

And to use dd to cut the interesting part

dd if=toto bs=1024 skip=1236 count=1

Tripwire is a nice security and data integrity tool useful for monitoring and alerting on any file change.
Tripwire can be downloaded here: http://sourceforge.net/projects/tripwire/

Tripwire create a database about audited file (with size, date of modification, hash...) and will periodically check if some modification happened.

When you install tripwire you have to create a security key. This key will be used to sign the policy files and to validate data.

- Edit the policy file. There, you will defined which files will be audited.
/etc/tripwire/twpol.txt
- Generate the key
/etc/tripwire/twinstall.sh
- Update the policy anytime you modify the policy (because tripwire store it in a binary signed file)
/usr/sbin/twadmin --create-polfile -S /etc/tripwire/site.key /etc/tripwire/twpol.txt
- generate the database
tripwire --init
- Now if you when tripewire to checj periodically you can invoke tripwire in a script from the crontab. For exemple, you can create the script
/etc/cron.daily/tripwire-check

#!/bin/sh -e

tripwire=/usr/sbin/tripwire
[ -x $tripwire ] || exit 0
umask 027
$tripwire --check --quiet --email-report

Every day reports will be generated in the directory /var/lib/tripwire/report/

Later, when a file will change on your system, you can update the database after you received a rapport
- Update database
/usr/sbin/tripwire --update --twrfile /var/lib/tripwire/report/rapport_name

Some useful commands for managing the postfix queue:
postsuper -d Number (remove a message)
postsuper -d ALL (remove all messages)
postsuper -r Number (re queue the message)
postsuper -r ALL (re queue all the messages)
postqueue -p (print the content of the queue)
postqueue -f (flush the queue, try to deliver all queued mail)