The Benefits of Enabling Timestamps in Your Command-Line History
While working at TrustedSec, I was issued a new company-furnished laptop to work from. While the Mac OS environment was useful, I found it useful to also setup an Ubuntu virtual machine. One reason is so I can have access to a Linux host that is very similar to the garden variety of Linux systems that I get to review during threat hunting and investigations. This blog post will walk you through setting up and enabling logging that I found extremely helpful not only for company work, but for projects and play too. Specifically, we will walk through setting history timestamps in the command-line history facility of the OS.
In my work, I often think about a command that needed to be run from the Terminal in a command-line manner and whether it was either useful, complicated, or had many steps to it, etc. So, if these commands needed to be recalled, we can do so using the well-known facility that has been around since the BSD editions of Unix: History. These tend to be very useful to recall what user-entered commands the OS recorded. To those of us that investigate suspicious computer activities, malware outbreaks, or breaches, the history records a wealth of key artifacts and activities. However, telling when it occurred might be tougher if the system is set to default (https://par.nsf.gov/servlets/purl/10213652).
tmillar@HP8200:/dev/shm$ cd /dev/shm tmillar@HP8200:/dev/shm$ touch altair deneb vega tmillar@HP8200:/dev/shm$ history 1 cd /dev/shm 2 touch altair deneb vega 3 history
By looking at other examples, we may be able to draw conclusions of when commands were issued to system by the person in control of the user ID. Please keep in mind that file timestamps are like boot prints in the mud: the most recent is the one you are most likely to make out to a good degree of certainty for each, Modified (mtime), Accessed (atime), or Changed (ctime). It is also important to note that Linux file system ext2/3 will not have Created (btime); however, this has since been implemented under ext4. Additionally, it’s important to understand that file timestamps can be manipulated or altered. When this occurs, it may throw off the examiner, unless they know where else to look or validate the time values being asserted to them.
Below is an example of this sort of situation. The contents of the /etc/passwd file are listed, redirected to the folder /dev/shm, and renamed to mycopy_passwd. The permissions to the file are then modified with the chmod command, and file stats can be checked to validate these permissions and timestamps.
$ cat /etc/passwd >> /dev/shm/mycopy_passwd $ chmod 744 /dev/shm/mycopy_passwd $ stat /dev/shm/mycopy_passwd File: '/dev/shm/mycopy_passwd' Size: 2418 Blocks: 8 IO Block: 4096 regular file Device: 18h/24d Inode: 3 Links: 1 Access: (0744/-rwxr--r--) Uid: ( 1000/ tmillar) Gid: ( 1000/ tmillar) Access: 2022-10-07 08:20:41.091102055 -0700 Modify: 2022-10-07 08:20:41.091102055 -0700 Change: 2022-10-07 08:24:45.491107440 -0700 Birth: -
The history record appears as this:
tmillar@HP8200:~$ history |tail -n4 859 cat /etc/passwd >> /dev/shm/mycopy_passwd 860 chmod 744 /dev/shm/mycopy_passwd 861 stat /dev/shm/mycopy_passwd 862 history |tail -n4
Next, we can use the cat command on the new file to see the contents. Rerunning the stat command on the file artifact allows us to see that the Accessed timestamp has been updated. From here, we can retrieve the history to make sure we get the complete listing of relevant entries. From what I see, everything that was entered into the command line is present in that listing. But keep in mind the history listing only shows that these happened in succession, and while it gives us the impression that each command happened right after the other, the stat output tells us that there was a delay of a few minutes between times of handling or accessing the file. The additional context given with the stat command gives us valuable perspective, especially when coupled with this history listing contents which contains the actual file(s) involved. Below, you can see the stat output followed by the history output.
$ stat /dev/shm/mycopy_passwd File: '/dev/shm/mycopy_passwd' Size: 2418 Blocks: 8 IO Block: 4096 regular file Device: 18h/24d Inode: 3 Links: 1 Access: (0744/-rwxr--r--) Uid: ( 1000/ tmillar) Gid: ( 1000/ tmillar) Access: 2022-10-07 08:27:18.515110812 -0700 Modify: 2022-10-07 08:20:41.091102055 -0700 Change: 2022-10-07 08:24:45.491107440 -0700 Birth: -
tmillar@HP8200:~$ history |tail -n8 859 cat /etc/passwd >> /dev/shm/mycopy_passwd 860 chmod 744 /dev/shm/mycopy_passwd 861 stat /dev/shm/mycopy_passwd 862 history |tail -n4 863 cat /dev/shm/mycopy_passwd 864 stat /dev/shm/mycopy_passwd 865 history |tail -n7
Now we have a perspective of when things happened based on the metadata timestamps that show when access occurred, at 08:27 local time, and when the entry within the folder, /dev/shm, occurred with the chmod taking place at 08:24 local time.
But what options are available when you don’t have command activity that can tell you when the actual commands took place outside of making inferences or examination of log records? What can you do if someone intentionally destroyed the files that you wanted to review after they were referred to in the .bash_history file that you needed to understand where they fit into the timeline? This gets at the heart of what we talk about next.
The modern-day Linux host can track not only the actual command parameters issued into the command line, but also keeps track of the time in which it is done. A system value referred to as $HISTTIMEFORMAT defines HOW the OS displays the commands and the time they took place. However, this is only available if the file gets saved to disk. If by happenstance where a series of commands were being issued and the power suddenly went out, there is a pretty good chance that edition of the .bash_history file will not be written to the disk and preserved. This means that only the prior session and its contents will be present when the system comes back on. However, chance occurrences such as power disconnections or other unplanned calamities are that: chance. Otherwise, the examiner may stand a chance to uncover history artifacts and the time they took place. One key consideration about the $HISTTIMEFORMAT is that after specifying an actual format string (eg., %F - %Z), the Linux operating system will record time values of each command entered in that user profile, in the most machine efficient way possible. Usually, this means in Unix Epoch Time. That is the value of time in seconds since midnight on January 1st, 1970, GMT. This is effective for keeping time tracking down to a resolution of one second, but it also offers us another feature we will discuss later.
Here we can set a format string of our history timestamp to give us the date and time in local time terms:
%F -%R
My personal preference is to have my history listings include the Unix Epoch Time, the local time, and time zone because it's convenient to be able to recall useful or complicated commands. The human readable output is useful as this allows the output from history to be ‘grepped’ by day or month, which is especially handy. Having the Unix time listed helps me understand what the relevant time in seconds was when commands were issued.
The following command history shows the order of commands used to install TimeSketch on a new Ubuntu host. In it, you can see the Unix Epoch Time as well as human readable with a TZ designation prepended to each command.
10 1665081386 2022-10-06 11:36-PDT curl -s -O https://raw.githubusercontent.com/google/timesketch/master/contrib/deploy_timesketch.sh 11 1665081407 2022-10-06 11:36-PDT chmod 755 deploy_timesketch.sh 12 1665081428 2022-10-06 11:37-PDT ls 13 1665081441 2022-10-06 11:37-PDT mkdir Tools/installers 14 1665081518 2022-10-06 11:38-PDT mkdir Tools/installers/timesketch 15 1665081528 2022-10-06 11:38-PDT mv deploy_timesketch.sh Tools/installers/timesketch/ 16 1665081532 2022-10-06 11:38-PDT cd /opt 17 1665081542 2022-10-06 11:39-PDT pwd 18 1665081556 2022-10-06 11:39-PDT sudo ~/Tools/installers/timesketch/deploy_timesketch.sh 19 1665082385 2022-10-06 11:53-PDT sudo docker-compose exec timesketch-web tsctl create-user parallels 20 1665082913 2022-10-06 12:01-PDT sudo docker-compose down 21 1665082936 2022-10-06 12:02-PDT ls 22 1665082939 2022-10-06 12:02-PDT cd timesketch/ 23 1665082940 2022-10-06 12:02-PDT ls 24 1665082946 2022-10-06 12:02-PDT sudo docker-compose down 25 1665082953 2022-10-06 12:02-PDT sudo docker-compose up -d 26 1665082961 2022-10-06 12:02-PDT history
Regardless of the chosen format style applied, the Unix Epoch Time format and Linux OS-dependent delimiters (eg., the #) are what will be written to the disk. Even if the history is cleared, whether maliciously or otherwise, those that were written will leave behind a discernable and useful pattern within the data structures: a whitespace character, followed by a pound sign (#), and then the Unix Epoch Time. From that, one may be able to formulate a regular expression search to identify matching data on disk. Time and time again, these have come to the surface when carving items out of unallocated space and have greatly helped me understand what happed in an incident.
The exhibit below shows you a portion of a history file as it is written to disk, in both the hexadecimal format along with the plaintext ASCII strings on the right.
$xxd .bash_history 00000000: 2331 3636 3530 3831 3137 370a 6869 7374 #1665081177.hist 00000010: 6f72 790a 2331 3636 3530 3831 3238 360a ory.#1665081286. 00000020: 7375 646f 2061 7074 2069 6e73 7461 6c6c sudo apt install 00000030: 2064 6f63 6b65 722d 636f 6d70 6f73 650a docker-compose. 00000040: 2331 3636 3530 3831 3331 300a 6869 7374 #1665081310.hist 00000050: 6f72 790a 2331 3636 3530 3831 3333 360a ory.#1665081336. 00000060: 1b5b 3230 307e 6375 726c 202d 7320 2d4f .[200~curl -s -O 00000070: 2068 7474 7073 3a2f 2f72 6177 2e67 6974 https://raw.git 00000080: 6875 6275 7365 7263 6f6e 7465 6e74 2e63 hubusercontent.c 00000090: 6f6d 2f67 6f6f 676c 652f 7469 6d65 736b om/google/timesk 000000a0: 6574 6368 2f6d 6173 7465 722f 636f 6e74 etch/master/cont 000000b0: 7269 622f 6465 706c 6f79 5f74 696d 6573 rib/deploy_times 000000c0: 6b65 7463 682e 7368 0a23 3136 3635 3038 ketch.sh.#166508 000000d0: 3133 3634 0a77 6869 6368 2063 7572 6c0a 1364.which curl. 000000e0: 2331 3636 3530 3831 3338 360a 6375 726c #1665081386.curl 000000f0: 202d 7320 2d4f 2068 7474 7073 3a2f 2f72 -s -O https://r 00000100: 6177 2e67 6974 6875 6275 7365 7263 6f6e aw.githubusercon 00000110: 7465 6e74 2e63 6f6d 2f67 6f6f 676c 652f tent.com/google/ 00000120: 7469 6d65 736b 6574 6368 2f6d 6173 7465 timesketch/maste 00000130: 722f 636f 6e74 7269 622f 6465 706c 6f79 r/contrib/deploy 00000140: 5f74 696d 6573 6b65 7463 682e 7368 0a23 _timesketch.sh.
This shows a relation between the prior exhibit on the basis of time, as it is recorded in terms of seconds since January 1, 1970 GMT. You can also clearly make out a structure for each record dealing with the time and recorded command.
This was produced when the following Bash lines were added to the /etc/profile file of the Ubuntu system.
HISTSIZE=7000000 HISTTIMEFORMAT="%s %F %R-%Z " if [ "$HISTCONTROL" = "ignorespace" ] ; then export HISTCONTROL=ignoreboth else export HISTCONTROL=ignoredups fi
A description of those options starts with the %s denoting Unix Epoch Time in terms of seconds, followed by a string that prints a numerical form of year-month-date, followed by a space. The time is reported in 24-hour clock form of hours-minutes-seconds with respect to the local time of the host. The time zone (TZ) designation is nearly last with a hyphen between the time and the TZ. For readability, I deliberately added a whitespace character so the TZ term and the beginning of the recorded command line will have a bit of separation between them. Individual taste in the format strings and variables can be found by the main pages for the interpreters (SYS V, csh, bash etc) you are using. You may also wish to add them to the individual profile files for the superuser ‘root’ account as well.
One thing to note before to adding this to the .bash_profile file, or into the global file, /etc/profile, is that the first time the user performs a reboot or logs out, logs back in, and checks the history, something unusual and confounding appears in the listing. It would appear that all of the history entries prior to the time that they defined the $HISTTIMEFORMAT value, all of them have the same exact time. This is a known issue and can be solved by backing up the prior history file contents. The next step is then clearing the history after setting $HISTTIMEFORMAT. That way you can have a record of all the command events applied, and they will have their own time values down to the second. Be sure to check this if you are applying it after using the host. That is why I like setting it up immediately after setting up a new Linux/Unix system or virtual machine.
In this article, we have discussed that even though the Linux history may not be completely certain and may not tell you the specifics when things were performed in a command line session. However, we went over a few ways to add timestamping to the history record entries and discussed ways that data can be used during computer security incidents or forensic circumstances. I hope you enjoyed reading through this and come away considering what you may wish to employ to help with your command history logging needs.
References:
“Use of Bash History Novelty Detection for Identification of Similar Source Attack Generation”, Jack Hance and Jeremy Straub, https://par.nsf.gov/servlets/purl/10213652