Security Raiders for UNIX Emergency Response

Remember: Now every step of the victim can now change the existing evidence or lead to the loss of sensitive information!

{{Initial response}}}

Objective: Overview of the emergency outline is initially determined before the judicial identification replication is performed.


Create a response kit


We investigate the system, you must execute commands with highly reliable programs, plus backup and repair, and create a toolkit is necessary. Even on the non-UNIX / Linux system, the creation of the toolkit should also be the first step in response.

First, we need to compile the response during the system of the corresponding architecture, and the compiler needs to consider system-compatible issues.

Usually we need tools as follows:

LS DD des file pkginfo


NetStat Pcat Perl Ps Strace

Strings Truss DF VI

Cat Kstat ifconfig chkrootkit

More Gzip Last W RM

Script Bash Modinfo LSMOD

Readers can add themselves according to their own needs, but a toolkit can only be used to complete the work of a particular platform, putting tools compiled with multiple platforms into the same tool package. It will be disordered. When you create a response toolkit on Linux, you can compile the source code with the GCC’s CSTATIC parameter, or use the LDD to check the dynamic connection library, set the library file directory on the toolkit storage medium, and copy all the dynamic connection libraries required by all tools. Finally set environment variables. This process is a bit similar to creating a Linux-based USB flash drive.


Get volatile data


Volatile data includes: the currently open socket, process list, RAM content, and non-linked file location. * UNIX Features: UNIX Allows the process to delete it when executed! Non-linked files are files that are marked as deleted when the process of accessing the file is aborted. When the system is turned off (normal shutdown or sudden power-off abnormal shutdown), the file marked as deleted will disappear. So before you find the file marked as deleted, you cannot shut down!


Perform trusted shell


Use our own response toolkit to load the media file system, Mount CT Auto / DEV / SDA1 / MNT / USB or

Mount CT ISO9660 / DEV / CDROM / MNT / CDROM Press CTRL + Alt + F1 to F6 to log in as root from the console. Be sure to distinguish between commands in the original environment variable and the same name of the current response toolkit, prevent potential binary Trojan attacks.


View users in the login system


[root @ AY4Z3RO FOO] # w

19:50:48 Up 43 min, 2 Uses, Load average: 0.00, 0.00, 0.00

User tty login @ iDLE JCPU PCPUWHAT

Root: 0 19:08? XDM? 11.10s 0.43s gnome-session

Root PTS / 0 19:08 1.00S 0.21S 0.01S W

Output Title Row shows the current system time, the system has run, the number of users currently logged in the number, the last 1 minute, 5 minutes, and 15 minutes. The user field displays the username of the current login. The TTY field shows the control terminal of the session. TTY indicates that the PTS / TYP can be represented by a network connection because x is a C / S mode application, so I am displayed as PTS in GNOME. . If you do not log in locally, there is an from field in the output, indicating the domain name or IP of the source address of the session. Login @ Displays the local start time of the connection. The IDLE field shows the length of time since the last process run. The JCPU displays the time used in all processes associated with TTY or PTS. The PCPU field displays the CPU time used in the current process in the WHAT. What columns displays the process of user currently run.


View System Process List


PS CEAF is used in Solaris, and PS CAUX is used in FreeBSD and Linux.

[root @ ay4z3ro foo] # ps auxuser pid% CPU% MEM VSZ RSS TTY Stat Start Time Command

Root 1 0.1 0.2 1356 496? s 19:07 0:04 init

Root 2 0.0 0.0 0 0? SW 19:07 0:00 [KEVENTD]

Root 3 0.0 0.0 0 0? SWN 19:07 0:00 [ksoftirqd_cpu0]

Root 4 0.0 0.0 0 0? SW 19:07 0:00 [kswapd]

Root 5 0.0 0.0 0 0? SW 19:07 0:00 [bdflush]

Root 6 0.0 0.0 0 0? SW 19:07 0:00 [Kupdated]

Root 7 0.0 0.0 0 0? SW

Root 11 0.0 0.0 0 0? SW 19:07 0:00 [kjournald]

Root 114 0.0 0.5 2108 1304? s 19:07 0:00 Devfsd / dev

Root 209 0.0 0.0 0 0? SW 19:07 0:00 [KHUBD]

Root 338 0.0 0.0 0 0? SW 19:07 0:00 [kjournald]

RPC 620 0.0 0.2 1496 520? S 19:07 0:00 [Portmap]

Root 636 0.0 0.2 1452 624? S 19:07 0:00 syslogd -m 0

………………… (omitted below)

The START field in the PS command output shows the time that the program starts running, which is helpful for detecting the attack time. Sometimes you can identify suspicious processes only through time. LINUX can also view the full command line parameters of the system running the process in the system through Strings CF / Proc / [0-9] * / cmdline, but this is not fully credible. Because the attacker does not even need to insert the kernel module, it can deceive us only in the encoding of the application layer.


Detect LKM rootkit


The kernel module is behind, what is more than this more stinking? Solaris, Linux and almost all UNIX supports LKM (LOADABLE KERNEL MODULES), which cannot be detected with ordinary ways, which brings great challenge to emergency response. For us, the solution is to find those LKM Rootkit, and are familiar with them, dissect them. Sometimes LKM rootkit is successfully loaded, but “exception” will appear in some details of the system, and may even make the system crash after a period of time. Also, LKM is active in Ring0 core state, but attackers tend to leave traces somewhere in the system, such as an attacker can automatically load his resettle kernel back door after each shutdown or reboot, may be rewritten. /etc/modules.conf or /etc/rc.local. kstat / ksec is a tool for detecting LKM, formerly for Linux, the latter for * BSD.

[root @ AY4Z3RO KSTAT] # ./kstat

USAGE: ./kstat [-i iff] [-P] [-P pid] [-M] [-m addr] [-s]

-i iff may be specified as “all” or as name (e.g. eth0)

Displays Info About The Queried Interface

-P Displays All Processes

-p pid is the process id of the queried task

-M Displays the kernel “s LKMS” Linked List

-m addr is the hex address of the queried module

Displays Info About The Module To Be Found At Addr

-s Displays info thing the system calls “The -s parameter is most useful, it shows the information of the system call entry, which can detect the two cores of the most popular KNARK and Adore on the market, but theoretically he can’t detect All LKM Rootkit. Kstat / Ksec Site: actually familiar with the kernel attack, knowing that kstat simply checks sys_call_table [] The way is now attacked.

Now LinuxForum Security version of the master Madsys has articles on phrack61:

Finding Hidden Kernel Modules (The Extrem Way) – Link: 434871 & Page = 0 & View = Collapsed & Sb = 5 & o = All & Fpart =


Detect open port and related process


[root @ ay4z3ro foo] # NetStat CANP

Active Internet Connections (Servers and Established)

Proto Recv-q Send-Q Local Address Foreign Address State PID / Program Name

TCP 0 0

TCP 0 0 Listen 908 / x

TCP 0 0 Listen 880 / SSHD

UDP 0 0 /

Active Unix Domain Sockets (Servers and Established)

Proto Refcnt Flags Type State I-Node Pid / Program Name Path

UNIX 2 [ACC] stream listenging 2753 756 / / /tmp/.font-unix/fs-1

… (hereinafter omitted)

On Solaris, HP-UX, AIX, FreeBSD, LSOF tools can be used to list all running processes and their open file descriptors, including regular files, library files, directories, UNIX streams, sockets, and more. If you just want to display the process of network socket:

[root @ AY4Z3RO FOO] # l l c

Command Pid User FD Type Device Size Node Name

Portmap 620 RPC 3U IPv4 2598 UDP *: SunRPC

Portmap 620 RPC 4U IPv4 2609 TCP *: SUNRPC (Listen)

SSHD 880 root 3u IPv4 2885 TCP *: SSH (Listen)

X 908 root 1u IPv4 2945 TCP *: X11 (listen)

Particularly need to pay attention to the strange process and the open original socket.


Looking for whether there is a illegal sniffer in the system


In order to achieve this, we need to check if the network card is in a mixed mode: [root @ ay4z3ro foo] # ifconfig Ci Eth0 Grep Promisc promises does not appear on all * NIX, can be judged by the LSOF + PS command Whether it is running a sniffer. Or through a third party tool, such as Antisniff.


Check / Proc file system


The survey is more meaningful in / proc / $ pid / directory: EXE link, fd subdirectory, cmdline file.

[root @ AY4Z3RO 880] # ls -al

Total 0

DR-XR-XR-X 3 root root 0 Sep 20 19:53 ./

DR-XR-XR-X 62 root root 0 Sep 20 15:07 ./

-r – r – r – 1 root root 0 Sep 20 19:54 binfmt

-r – r – r – 1 root root 0 Sep 20 19:54 cmdline

LRWXRWXRWX 1 root root 0 Sep 20 19:54 CWD -> //

-r ——– 1 root root 0 Sep 20 19:54 environ

LRWXRWXRWX 1 root root 0 Sep 20 19:54 EXE -> / usr / sbin / sshd *

DR-X —— 2 Root root 0 Sep 20 19:54 FD /

-r – r – r – 1 root root 0 Sep 20 19:54 MAPS

-rw ——- 1 root root 0 Sep 20 19:54 MEM

-r – r – r – 1 root root 0 Sep 20 19:54 mounts

LRWXRWXRWX 1 Root Root 0 Sep 20 19:54 root -> //

-r – r – r – 1 root root 0 Sep 20 19:54 Stat

-r – r – r – 1 root root 0 Sep 20 19:54 Statm

-r – r – r – 1 root root 0 Sep 20 19:54 status

The EXE link allows us to recover the deleted file, as long as these files are still running. To obtain a backup of the “Deleted” executable, you only need to create a copy on the file system using the CP command. By checking the FD subdirectory, you can identify all files opened. If you have a programming in the UNIX environment, it is easy to find that you read and write a file or open a network connection. The content of the cmdline file is the full command line of the process. The following statement is the attacker’s spoofing means, STRCPY (Argv [0], “Any_String”); so that this file shows a description, even so, we still need to check this file.


Get all files creation, modify, and access times


LS CALRU> / MNT / USB / Access




Get system logs


Most UNIX logs are different in / var / log and / var / adm directory, and the specific location of various UNIX derived system logs is different. Before this, it is necessary to understand the log storage location for a particular system. More important binary log files:

UTMP, use W tools to access;

WTMP, access to Last tools;

LastLog, use LastLog tools;

Process accounting log, use ASTCOMM tools to access

Common ASCII text log files:

Apache Log – / VAR / LOG / HTTPD / Access_log;

FTP log – xferlog;

Command history file;

/ var / log / messages;

Check the /etc/syslog.conf and other daemons profiles to determine the location of the rest of the log.


Get important profiles


Check each configuration file to find the back door position, unauthorized trust relationship and unauthorized user ID. / etc / passwd, find unauthorized user accounts and permissions. The primary intruder will add UID = 0 users, and some people will also change a normal account in the system in the system to the login to get the shell execute command, and then he can pass the KSH or other placement in a SuID. Get rootshell./etc/shadow immediately, make sure that each user has password authentication; of course, an attacker adds an MD5 hash to her account. It is also very simple.

/ etc / groups, find the upgrade of permissions and expansion of access.

/ etc / hosts lists local DNS entries.

/etc/hosts.equiv, check trust relationship.

~ / .rhosts, check the user’s trust relationship, “++” This very well-detrimental back door believes that everyone knows.

/etc/hosts.allow && /etc/hosts.deny Checks the rules for TCPWrapper.

/ etc / rc *, check the startup file.

CRONTAB file listing the plan event.

/etc/inetd.conf, list the service listening to the port.


Dump system RAM


Mainly from the system transfer / proc / kmem or / proc / kcore file, the file contains the contents of the system RAM in a non-continuous manner.

{{look deep into}}


Check system log


Unix has a lot of logs that provide important clues for emergency response. Most of the log files are located in public directory, usually / var / log, / usr / adm, / var / ADM, and some logs are located in disabled / etc directory. Please refer to the current operating system documentation. The syslogd daemon provides a very powerful log function, such as loading a kernel module registration, its profile is /etc/syslog.conf,

Usually it provides the most useful logs are: Messages, Secure, Syslog. Each row in syslog.conf contains three fields: The Facility field represents a subsystem that generates the log file; Priority field indicates the severity level of the event; Action field indicates How to record the log, it provides the ability to remotely record. The TCP Wrapper log also uses the syslog record, which may have remote login of Telnet, SSH, FTP. These logs

There are many valuable entries: Try the time dates, host name, access service type, and source IP address. Other network logs, for example, web, ftp, and sql usually provide more detailed information. Apache Default Log In / usr / local / apache / logs, the most useful log is Access_log, and SSL_Request_Log, SSL_Engine_log can also provide valuable information. It may contain scan recording before the attack.

The su command log records the action of executing the su command: time date, successful or not, terminal device, user ID. Some UNIX has a separate Su log, some saved in syslog.

Log in the user log: UTMP or WTMP file saves information about the user currently logging in to the system. This file is different from each UNIX version.

The name and storage location are different. The basic information saved is the username, the terminal used to log in, and the time to log in. The file is stored in binary format.

Query UTMP, WTMP files should use the appropriate client, such as W, WHO, Finger, Last. Search success, failure and username unknown landing entry.

The Cron log records the contents of the timing job, usually in the / var / log / cron or the default log directory called cron file. Process bookkeeping, if the system exists ACCT or PACCT log file, you can use the LastComm or AccTcom command to view. This log is binary. Shell History: [Root @ AY4Z3RO FOO] # ipes ~ / .bash_history

If .bash_history is linked to / dev / null file, or $ histfile in the environment variable, $ HistFileSize two variables is 0, then someone is illegally active.

Most intruders will modify or delete logs. Although it is theoretically to plant LKM Rootkit almost no traces, in the actual invasion, the good work is actually a small project, not only relying on the intruder The degree of familiarity, and it is easy to appear when the content is handled too much. For example, when you just get ROOTSHELL, Unset Histfilesize, I forgot to restore, leaving a trace. There are still many examples, and the log clearance tool is dead. It only clears a predefined item, although you can also modify the source code, but you can’t randomly strain. The most insurance method is manual labor, which increases the burden of intruders. For lazy, it is not enough to have a variety of things for the system. Therefore, check logs are very important to emergency response. ==============

Execute keyword search


Whether it is an emergency response to the operating system, keyword search is part of the process. For a specific event, there may be some ID, phrase is closely related to this incident, and more information can be found in executive keyword search. Keywords can be a long ASCII string, including attacker back door password, user name, MAC address, or IP.

Example: Search all files containing the “AY4Z3RO” string in the entire file system:

[root @ ay4z3ro foo] # GREP CR CI AY4Z3RO / STRINGS command is used to display the printed characters in the file. Hash). The Find command is used to find any file names that match the regular expression. Example: Search “…” files or directories in the entire file system:

[root @ ay4z3ro foo] # Find / -name “. . .” CPRINT

The feature that can also match can also include: Modifying the access time, the file owner, the string of the file, and the string of the file name, etc. The -EXEC options in the Find command are combined with grep, Strings, which reflects the genius temperament of UNIX, which is very convenient.


Determine the emergency time


* If there is IDS, make sure the IDS system time is consistent with the victim time.

* The files created and changed before and after the emergency in the system may have surprises.


Restore deleted files and data


This process requires an operator to understand the architecture of the UNIX / Linux file system. It does not intend to introduce the data structure of the file system and its code, but a simple explanation, deleting a file is actually just put the inode node Part of data and pointer are set to 0, and the contents of the data block to which the node points to by the node are not deleted before the next data overlay. To resume the deleted file, you need to use the information of the inode node to rebuild the file size and data block list. Find file node information:

[root @ AY4Z3RO FOO] # lrs ci /tmp/x.d

82241 / TMP/X.D

Indicates that /tmp/x.d is located in the 82241 node.

TCT (The Coroner “S Toolkit is a very useful kit, where ICAT can view all the contents of the file on the specified node.

[root @ AY4Z3RO TCT] # ./icat / dev / hda5 82241

If the program is still run, you can use the lsof command to reference the Node column to find node information.

Restore a file only:

[root @ AY4Z3RO TCT] # ./icat / dev / hda5 node> Some.Recovery

There are also many useful tools in TCT, examples:

[root @ AY4Z3RO TCT] # ./ils ca / ??dev / hda5 grep “501”

The above command line identifies all deleted files related to the UID = 501 user.

TCT link:


Check special files


SUID and SGID files:


[root @ AY4Z3RO /] # Find / -Type F (-perm -04000 -o -perm -02000) -Exec ls -lg {} Find out that those who have not contains S-bit, unusual or being placed in one Strange place S-bit program. The / bin / KSH is renamed it after the / TMP (777) directory is a back door.

Not common and hidden files and directories:


The files beginning with “.” In the UNIX system are hidden. If the LS does not add “-a” parameters, it will not appear in the file list.

Renaming the catalog with hidden characters is a hacker skill, such as some file names: “. ^ T”, “…” (note that there is a space behind three points) …

This can deceive a lot of system administrators.

How to do it? as follows:

[root @ ay4z3ro foo] # ls ca “Cat Ctve”

The parameter of the CAT command allows it to display non-print characters, displays tabs and placed one $ at each end, so the above directory will appear as: “. ^ t $”, “… $”.

To enter the previous directory, press CTRL + V before t, not using ^ symbols, and enter a directory is: [root @ ay4z3ro foo] # CD “…”



If an attacker has a fairly skilled system management skill, stealing the column, doing a handfoot in the configuration file is very easy. For example, modify or add / etc / services, /etc/inetd.conf, but for skilled system administrators, such a back door is easily discovered. There is also some Exploit by-products, such as the SADMIND RPC remote overflow of Solaris2.6, the default attack program is overflow in /etc/inetd.conf

Add a forged Ingreslock service to bind / bin / sh bind at 1524 ports. Generous attackers usually do not pay attention to this problem.

Start the file:


The above inetd.conf is such a file, in addition to this Cron file, directory / var / spool / cron, / usr / spool / cron to save the cron job for different users. The files in this directory are named by the user account, and the tasks are running in this user privilege. The root file under this directory should be our concern. Some people like to start the Trojan in the middle of the night to run a bindshell, and then close the open port in a few hours.

/ etc / rc * is a start-up automatic run script and is often used by attackers. The rest of the user starts files, such as login, profile, .bashrc, .cshrc, .exrc may be inserted into Troy statement, and it has been used in’s incident.

/ TMP directory:


/ TMP directory authority is 777, which is where attacker is often used, and many hacker tools also use this directory to store intermediate files. If the attacker is not careful,

It will leave the tail here.


Check user accounts and groups


Some accounts are set for the system. There is no shell, and the attacker may take this. People who have just learned Unix invasion may be like this:

Echo “AYA: x: 0: 0: Intruder !!: /: / bin / sh” >> / etc / passwd, more poor guys destroyed the passwd file because of a “>” and got itself I can’t log in? I have seen such guys. I don’t have the truth, but I happen to my own Linux, so I didn’t cast a big mistake. Sophistication will not do this, they may pick up a bunch of users from a bunch of users, and then account for them, the remote legal login gets the shell and then upgrade to root. I believe this problem is very good for us. solve.


Identify illegal process


Searching services and running process-related binary should check, view /etc/inetd.conf may find legal services in legal port listening, but the binary files may be replaced, so make sure It is not ROOTKIT (LRK4, LRK5 …)

[root @ ay4z3ro Tool] # ./chkrootkit chkrootkit is a tool for checking integrity. If you have used TripWire, you can check it out. Or use rpm itself MD5 checksum function. ============

Exploration system weaknesses


Check the version of each service, application, kernel and patch, and look up the system weaknesses on Bugtraq, find the system weakness, mining potential and possibly ignored vulnerabilities. This requires responsive personnel with a professional intruder’s related skills! Trying to play an intruder can also explore how the other party enters the system.


Analysis trust relationship


First, the trust relationship in UNIX once becomes the weakness of the attacked. Second, if the relationship is utilized, the range of victims will be expanded, and the trusted system is also considered unsafe and simultaneously incorporates the range of response objects.


Analysis hacking tool


If it is very fortunate, the invaders left or to restore the tools and code used in the active process in the event, and they can further analyze them. If you are the source code, you can read it directly (of course, the source code of LKM Rootkit is best, huh). Unskilled guys don’t even change the original tool, such as the Sniffit name, so it is better to deal with.

If we get a copy of the binary file that is running the process, we can use the GDB and other debuggers to disassemble, track debugging. But if a very level of attacker compiles his procedure:

[root @ AY4Z3RO EVIL] # GCC CO4 Evil.c Co Evil Use the -O4 parameter to optimize the instruction, then use Strip to remove symbol information in the binary:

[root @ AY4Z3RO Evil] # Strip ./evil, our work will become very difficult. The file command can display the type information of the file, whether it is too equal to Strip. The strings command can be used to display the ASCII string in the executable, such as a line, error handling message controlled by the Printf () statement, and the return information of the default -H parameter, etc. In addition, it is possible to obtain the file name used before the function, the variable name, and compile. Create the compiler version of the file, etc., the online search is possible to find the source code of the tool by these keywords.

Also we can dynamically analyze the binaries and track system calls with STRACE tools. Strace displays file access, network access, memory access, and many other system call information generated when executed. Usually call us by observing the key system to determine what this program is made. It is also possible to restructuring the operation of the file. Strace provides us with great convenience. In the entire response process, we can also use it to do a lot.

{{ºó »°}}}

Emergency response does not need to completely follow the fixed mode, and the ideas can be moderately played. If the opponent encountered is a high level of attackers, it should be appropriately changed appropriately. For safety, the final reload system may be necessary.

Write to the majority of “engage in machinery”:

Even if someone feels that he invaded skills, but in fact, the defense is always more or less, some of the strategies that you expect, a little horses and tiger may make you embarrassed. Even if you think that all logs on the target system have been cleared, but the front-end router or IDs, there may be no “handling” or even the tracking record you have never been recognized, for its own security, find a quick line, use multiple It is necessary to start a real attack. Can’t expose yourself from “start” to “end”!