April 26, 2018
Malware: Linux, Mac, Windows, Oh My!
Written by
Kevin Haubris
Incident Response
Incident Response & Forensics
Malware Analysis
While going through APT write-ups, I’ve been noticing a lot of focus on detecting Windows malware, so we will skip over that. One thing that I haven’t seen much of online, though, is how to hunt for adversaries on Linux systems. For that reason, this blog post will be all about how you can look for binaries on your Linux servers that shouldn’t be there, or other ways to maintain access. Keep in mind these may have some false positives, especially if you install a lot of code from source, but it should narrow it down to what binaries you can focus on. I will try to explain what each of these checks do as I write them up. If you’re a Linux incident response person there probably isn’t anything new here and I’m probably missing something, but I’m trying to cover the basics and things I know are common.
Verifying Binaries
One thing to check is to make sure no binaries running were modified. This type of malware could be backdoored versions of sshd to allow a specific password to be used to connect to the system, or even a modified version of some binary, that runs as root with added value that just listens with a raw socket for a trigger packet. For this we are going to go through a Redhat and Debian version. Finding binaries that don’t belong- find /proc/*/exe -exec readlink {} + | xargs rpm -qf | grep "not owned"
- find /proc/*/exe -exec readlink {} + | xargs dpkg -S | grep "no path"
- find /proc/*/exe -exec readlink {} + | xargs rpm -qf | xargs rpm -V
- find /proc/*/exe -exec readlink {} + | xargs dpkg -S | cut -d: -f1 | xargs dpkg -V
Fig. 1 - Finding and verifying binaries[/caption]
Verify All Package Files
Another thing that should be done is to make sure none of the binaries that belong to packages were modified. This takes a lot longer than just verifying the ones running but is worth it if cron jobs or anything else is set up to run it at a specified time. Verify all- rpm -Va
- dpkg -V
Fig. 2 - Verifying package files[/caption]
Reading the Results
The output should show any binaries that belong to packages, calculate the hash of the binary, and compare it to the one saved when the package was installed or updated. The below output is for Redhat based systems. Debian based systems with dpkg don’t verify a lot of these, so only “5” is shown if the binary was modified. S file Size differs M Mode differs (includes permissions and file type) 5 digest (formerly MD5 sum) differs D Device major/minor number mismatch L readLink(2) path mismatch U User ownership differs G Group ownership differs T mTime differs P caPabilities differCheck for RAW Sockets
Something we’ve been seeing more often is RAW socket backdoors. These listen for an incoming packet and trigger an event. One example is that the recent write-up on the “Chaos” backdoor links to the write-ups will be below, along with an example that popped up while searching for a raw socket backdoor on github. For this check we are just going to see what processes are using RAW sockets. There are not a lot of common programs using them, so this could narrow down what processes to look at if you think you were compromised. Check for binaries with raw sockets listening- netstat -lwp or ss -lwp
- lsof | grep RAW
Fig. 3 - Checking for raw sockets[/caption]
Checking for Possible Injected Memory
This one on its own will have all sorts of false positives. RWX memory (Read Write and Execute) is used by a lot of programs, most of which are interpreted languages, so things like python and java, or things using any libraries that parse scripts, will have this and it will be normal. If you find a lot of entries of RWX memory and that process isn’t python or java, you should probably look a little closer at it. This command will list the process ids with RWX memory. Things like gnome will be normal as well, but things like cron or any other normal process shouldn’t have any. Command to find pids- grep -r "rwx" /proc/*/maps | cut -d/ -f 3|uniq -c | sort -nr
Fig. 4 - Checking for possible injected memory[/caption]
Check for Modified PAM Modules
One common backdoor is inserting or replacing a PAM module for authentication. This can allow for remote access and also allow for an attacker to su to root from any user. This backdoor also doesn’t care about changes to /etc/passwd so all the original passwords and changed ones will still work; however, it has the actual password the attacker will use embedded into it. This is, in my opinion, a really dangerous type of backdoor due to the type of access it gives. You can use normal protocols with legitimate login entries, so there is obviously no malicious network activity. Verify PAM modules- find /lib64/security/ | xargs rpm -qf | grep “not owned”
- find /lib64/security/ | xargs rpm -qf | grep -v "not "| xargs rpm -V
[caption id="attachment_14075" align="aligncenter" width="726"]
Fig. 5 - Checking for modified PAM modules[/caption]
Last but Not Least… SSH Access
An extremely simple way to keep access that doesn’t require dropping a binary is simply adding a ssh key into the authorized_keys file for a specific user and allow the attacker to ssh in like a normal user. This is also one of the hardest methods to detect because you need to figure out if the ssh keys are legit or malicious, which requires users to verify that only their keys are in that file. An attacker could also just steal the key of a user if they were compromised before. List .ssh folder for all users- cat /etc/passwd |cut -d: -f 6 | xargs -I@ /bin/sh -c "echo @; ls -al @/.ssh/ 2>/dev/null"
Fig. 6 - listing .ssh folders[/caption]