Malware Analysis is for the (Cuckoo) Birds – Working with Proxmox

This post will be on how to setup and modify Cuckoo to work with a non-supported hypervisor, Proxmox.

Proxmox VE is a complete open-source platform for all-inclusive enterprise virtualization that tightly integrates KVM hypervisor and LXC containers, software-defined storage and networking functionality on a single platform, and easily manages high availability clusters and disaster recovery tools with the built-in web management interface.

The enterprise-class features and the 100% software-based focus make Proxmox VE the perfect choice to virtualize your IT infrastructure, optimize existing resources, and increase efficiencies with minimal expense. You can easily virtualize even the most demanding Linux and Windows application workloads, and dynamically scale-out your computing and storage as your needs grow ensuring to stay adaptable for future growth of your data center.”

Proxmox is built on the Debian platform. They provide a bootable ISO image, which can either be burned to a disc or better yet to a bootable USB flash drive. I used Ubuntu and its “Startup Disk Creator” to create the bootable USB drive.

Fig. 1 – Find USB Startup Disk Creator

Select the Proxmox ISO by clicking other and using the file dialog to navigate to the downloaded ISO. Next, select the Drive for which to copy the ISO. Lastly, click “Make Startup Disk”. This will take a few minutes but will prompt you when it is complete.

Fig. 2 – Startup Creator Main Dialog Screen

Boot the computer from the USB Drive to install Proxmox. The installation is straightforward and pretty quick. Follow the instructions on their website. The only issue I came across was the partitioning of the drive, as I used a small drive and quickly ran out of space.

 

Installing Cuckoo

The installation of Cuckoo is almost identical to the installation on Debian, as Proxmox is a variant of Debian. See Cuckoo Installation Notes for Debian for the full installation instructions. Here, I will focus on the additional steps needed to get Proxmox and Cuckoo to play nice with each other. Also, some of Cuckoo’s features are not supported as of this writing. I.E. Memory dumps for analysis through Volatility. I believe this can be done, but I have not invested the time as of this writing.

 

Setup Requirements:

The following commands are to be used in addition to the commands from Cuckoo Installation Notes for Debian.

pip install proxmoxer

Proxmoxer is a wrapper around the Proxmox REST API v2

pip install requests

Requests is an elegant and simple HTTP library for Python

sudo apt-get install qemu-kvm libvirt-bin Ubuntu-vm-builder bridge-utils

qemu-kvm -> Is already installed as part of Proxmox installation. Package pve-qemu-kvm

ubuntu-vm-builder -> Builds virtual machines from the command line

bridge-utils -> Utilities for configuring Linux Ethernet Bridges, normally already installed.

python-libvirt -> Provides an application written in Python with the ability to call the interface supplied by the libvirt library, manage virtualization capabilities. This functionality is provided by Proxmox. Installing this package will uninstall proxmox-ve and pve-manager. These two packages are the main functionality of Proxmox and the web interface. If you install the python-libvirt package, do the following to reinstall proxmox-ve and pve-manager.

apt-get install proxmox-ve pve-manager

For me, the sources.list was updated and I needed to modify that to get the correct package lists.

libvirt-bin -> Used to interface with different virtualization technologies.

 

Cuckoo Installation:

sudo adduser tweety

adduser -> Creates a new user. This example used the name tweety.

Proxmox logs in as the tweety user and does not have sudo by default.

apt-get install sudo

 

Configure Cuckoo

Cuckoo configuration is mostly the same as before, except: edit the cuckoo.conf, setup of the new VLAN and add the new configuration file, proxmox.conf.

Cuckoo.conf

Modify the machinery line to run proxmox.

Fig. 2 – cuckoo.conf Example Selecting Proxmox Hypervisor

Setup a second network interface that was bridge. Figure 3 demonstrates the network configuration. Modify the file /etc/network/interface to add the new bridge. A good resource for networking is here.

Fig. 3 – Proxmox Host Network Configuration

 

auto vmbr1

Instructs the system to bring this interface up on boot

iface vmbr1 inet static

Sets the interface to have a static IP address

address 192.168.56.1

Assigns the IP address of 192.168.56.1 to this interface

netmask 255.255.255.0

Mask used to divide the network into subnets. 255.255.255.0 allows 255 IP for this subnet. The subnet is 192.168.56.1 – 192.168.56.255

bridge_ports = enp8s0f0

Assigns the bridge to interfaces. This could bridge multiple network cards or none. In the case where the analysis does not want any traffic to leave the system, leave this line out. In this case, the cable is not connected to the spare interface enp8s0f0.

Alternatively, the network configuration can be completed using Proxmox Web UI. Highlight the Node name on the left, expand the System, and highlight the Network. Click on Create, then Linux Bridge.

Fig. 4 – Proxmox UI Network Setup

A new dialog window will be displayed. In this window, add the IP address for the Host, for this example, 192.168.56.1. The subnet will be 255.255.255.0. Check the VLAN aware box and fill out the Bridge ports if tying to interfaces, otherwise just leave blank.

Fig. 5 – Proxmox UI New Linux Bridge Configuration

The network settings can be viewed, edited, or deleted at any time from this page.

Fig. 6 – Proxmox Dashboard Host Network Configuration

Adding the Guest VM to a specific network. Highlight the VM on the left pane. Select Hardware and double-click Network Device. Add the created Network to the Bridge field.

Fig. 7 – Proxmox Guest Network Configuration

The configuration file, proxmox.conf needs to be added to the /home/tweety/.cuckoo/conf/ directory. This configuration file was originally based on virtualbox.conf, but has been modified to work with Proxmox REST API.

Fig. 8 – Example proxmox.conf
Sections of the proxmox.conf file 

The sections mirror those of the other machine configs. The first line is the name of the machine in square brackets “[“ and “]”

[proxmox]

            Name of the hypervisor.

username = tweety

password = <the password for the user tweety>

The username and password to the Proxmox GUI. This is used to log into Proxmox API. Currently, the password is in clear-text, so be sure to create a user with only privileges to start stop the sandboxes.

host = pveone

The Host will be the name of the server running the GUI. Usage of the IP address or localhost should also work.

nodename = pveone

Name of the node configured in Proxmox. Located below the Datacenter as shown below.

Fig. 9 – Proxmox Node Name Location
machines = kali2018, cuckooOffice

The machines are a comma separated list of the different VMs available to use. These names must match the later sections. IE. cuckooOffice muse have a section that starts with [cuckooOffice]

Virtual Machine Sections

[cuckooOffice]

Starts the virtual machine section and must match that in the machines tag above. This does not have to match the name of the VM.

label = cuckooOffice

the label must match the name of the VM. The label is used to identify the VMs unique ID. Proxmox uses the ID rather than the name. Both ID and name are shown in Figure 47.

platform = windows

The operating system of the Guest. I.e. windows, linux, or Darwin

ip = 192.168.56.99

IP address of the Guest. The Guest must have this IP address or the Cuckoo server will not be able to transfer the malware sample to the Guest VM.

snapshot = 

Cuckoo requires that each VM has at least one snapshot. If this is left blank cuckoo will revert to the last snapshot, otherwise this should be the name of the snapshot to revert too. As an example, if snapshot is left empty cuckoo will revert to the snapshot named Agent. If snapshot is set to InstallWin7 then that snapshot will be used.

Fig. 10 – Proxmox Snapshots
interface = vmbr1

The name of the VLAN or bridge created above.

resultserver_ip = 192.168.56.1

resultserver_post = 2042

The resultserver_ip and resultserver_port will match the IP and port configured in the cuckoo.conf and must match the static IP assigned in the VLAN/Bridge configured above.

Fig. 11 – cuckoo.conf Example

 

Cuckoo Code Modifications

Newer versions of Cuckoo have incorporated parts of the configuration files into the Python code itself. This stops the ability to just add a new machine.py file. However, this can be bypassed by adding the following to the /home/tweety/venv/lib/python2.7/site-packages/cuckoo/common/config.py

Fig. 12 – Cuckoo Code Modification config.py

Since the configuration file was modeled after the Virtualbox file, this section was originally copied from the Virtualbox section. As the items in the configuration file changed, those changes need to be reflected here. These values will be used in the proxmox.py and will be overwritten by values in the proxmox.conf file. Be sure to include the “__star__” line or else Cuckoo will error.

Fig. 13 – Cuckoo Error if “__star__” is missing

New Machine File

Each individual hypervisor will have a <machine>.py file located under /home/tweety/venv/lib/python2.7/site-packages/cuckoo/machinery/. Each <machine>.py file will be named the same as the configuration file ie. proxmox.conf and proxmox.py. The machine files are inherited from a base class, but leave you to implement the functionality such as start, stop, reverting a snapshot, and memory analysis. For simplicity, the use of the Python module promoxer is used to interact with the Proxmox REST API. This took some trial and error, as the modules did not work as the documentation explained. The following show the proxmoxer commands to retrieve specific hypervisor information.

proxmox.nodes().get(‘pveone/qemu’)  => list of vm’s in the the node ‘pveone’
proxmox.nodes().get(‘pveone/qemu/100’) => list of options for the VM 100
proxmox.nodes().post(‘pveone/qemu/100/status/start’) => Start the VM
proxmox.nodes().post(‘pveone/qemu/100/status/stop’) => Stop the VM
proxmos.nodes().post(‘pveone/qemu/100/snapshot’, snapname=’testing’) => snapshot the VM and name it ‘testing’
proxmox.nodes().post(‘pveone/qemu/100/snapshot/testing/rollback’) => revert the snapshot to the ‘testing’ snapshot

 

The source code for the proxmox.py is located here. The main methods that where modified are restore, start, and stop. Each of these methods used the proxmoxer commands listed above to implement the functionality. The key/value pairs from the proxmox.conf file are accessible through the self.options.proxmox variable. The initialize_check is called first to verify and set up the environment. This section is used to verify the configuration file input and to create the connection to the Proxmox REST API. The API is then used to determine the available VMs. It is possible to have more VMs in Proxmox than registered with Cuckoo.

 

Subscription Message Removal:

Proxmox is free and fully functional but they offer a subscription. If you use it, I would recommend migrating to their subscription service. After all, this is how they are able to provide the free version for us to use and enjoy. However, for a trial period the Nag message is a minor annoyance. To remove the Nag message, edit the /usr/share/pve-manager/js/pvemanagerlib.js file. Comment out lines 849 – 867. As shown in Figure 52. Then add the orig_cmd(); }.

 

Fig. 14 – Proxmox NAG message Removal

Cuckoo Usage

Now Cuckoo should be set up to run the new machine, proxmox. The actual usage of Proxmox is independent of the machinery. See Blog post Malware analysis is for the (cuckoo) birds.

 

Scott Nusbaum

Author: Scott Nusbaum

Over 14 years of experience in software development and 10 years in Information Security. Experience range from Material Handling, Banking, to the Defense Industry.