Recoving tmpfs from Memory with Volatility

In this blog post I will introduce a new Volatility Linux plugin, tmpfs, and discuss its uses and implementation. The purpose of this plugin, which can currently be found here, is to reconstruct any tmpfs filesystem contained within a Linux memory capture and fully recover it to disk. 

What is tmpfs?

Tmpfs is a Linux filesystem whose contents reside only in memory. This means that files and directories inside of tmpfs mounts are never written to the local disk and that once a tmpfs mount is unmounted, the entire filesystem is “gone”.  

Why is tmpfs interesting?

Tmpfs is interesting from a forensics perspective for a few reasons. The first is that, in a traditional forensics scenario, the investigator expects that he can shut a computer off, images its disk(s), and get back the filesystem at the time of when the computer was running. With tmpfs, this is obviously not true…

The second reason that tmpfs is interesting is because of how it is used by Linux distributions as well as by attackers and certain malware samples.  On many Linux distributions, the /tmp directory is mounted as tmpfs so that the directory will be cleared on reboot. This helps speedup the boot process as the init code does not need to unlink every file and directory under /tmp, but it also means that the traditional (disk only) preservation process will miss any files stored in /tmp. This directory often contains scratch data and install files for many applications as well as for rootkits/malware that try to be agnostic to the distribution they are being installed on. 

Linux also uses tmpfs to implement shared memory through /dev/shm. While recovery of this directory may help in recovering IPC data, its main purpose related to forensics & IR is that it is often used as a scratch directory by attackers to download files, compile programs, and to store the output of commands and malware hooks.  Since the filesystem goes away on reboot, attackers know they can frustrate and/or defeat traditional forensics investigations using this method.

The third reason that tmpfs is of interest is that Linux live CDs use tmpfs as the underlying store for all changes made to the filesystem after the machine boots, since obviously it cannot write back to the CD. If you are interested in reading about memory analysis of live CDs, I did a presentation and white paper on it at Blackhat DC 2011 ( slides | paper ) so I will not recap it all here.  The paper also discusses much of the kernel internals described at the end of this blog post.

Finally, while still being researched, it has been discovered that many Android applications use a tmpfs instance created by the Android runtime to store “interesting” runtime data and state.  The tmpfs plugin has already been tested on a number of phones using Volatility’s Android support and the data was verified to be fully and correctly recovered on all devices.  For the latest details on this, please contact Joe Sylve (@jtsylve).

How do you use the plugin?

The plugin requires two steps to work. First, you run it with the ‘-L’ option to list the tmpfs filesystems within the image:

python vol.py --profile=Linuxthisx86 -f /root/lime-tmpfs linux_tmpfs –L

The output of this invocation on my test VM looks like this:

Volatile Systems Volatility Framework 2.1_rc3

1 -> /lib/init/rw

2 -> /dev/shm

Now, let us say I want to recover the contents of /dev/shm, so I then run the plugin with the –S option of ‘2’ and an output directory:

python vol.py --profile=Linuxthisx86 -f /root/lime-tmpfs linux_tmpfs -S 2 -o outputdir/

When this command is finished, the entire filesystem of “/dev/shm”, including all files, directories, and metadata (atime, mtime, permissions), will be replicated to the “outputdir” directory.  In a real forensics investigation, “outputdir” should be on an external drive or a separate mount point that can be remounted read-only after the filesystem is recovered.

Limitations

When recovering the filesystem from memory to a disk there are a few limitations that investigators should be aware of:

1) While the script does preserve the modified and accessed times of the recovered files and directories, it has no method in which to preserve the create times. This is because the Python os.utime function only supports modifying the accessed and modified times. In order to change the create times on a standard Linux filesystem, the script would need to interact with something along the lines of debugfs, which can be error-prone and only supports the Ext family of filesystems.  Similar burdens are faced on Windows and Mac filesystems as well.

The result of this limitation is that the create time of all entries in the filesystem will actually be the time that the Volatility plugin created them. If enough interest is shown in the plugin and this limitation, I can easily extend the plugin to support writing out a mactime file with all the correct information.

2) The owner/group information is not preserved for recovered files. This may be added to future versions of the script, at least if the output filesystem is Linux based.

3) There is currently no support for recovering deleted files within tmpfs stores.

How does the plugin work?

Now that we have explained the purpose of the plugin, it is time to dig into some Linux kernel internal in order to explain the plugin's operation. If you are not a programmer or not interested in kernel internals, you can freely skip to the end of the post.

For the plugin to work it needs to be able to the following:

1) Find and list all tmpfs mount points in the memory image

2) Recursively enumerate and reconstruct the entire filesystem for every tmpfs instance

3) Record the needed metadata of each file and directory and update the recovered files on disk to match it

4) For every file found, recover the file contents and write them to disk

1) Finding all tmpfs mount points

 The tmpfs mount points are found by leveraging the capabilities of the mount plugin (volatility/plugins/linux/mount.py). The mount plugin works by walking the mount_hashtable  hash table, which stores a vfsmount structure for every mount point. This is the same operation the kernel does at runtime to populate the /proc/mounts file.  The tmpfs plugin then filters the vfsmount structures to only those representing tmpfs mounts.  This is all that is needed to implement the ‘-L’ option of the plugin.

 2) Recurse the filesystem

 Each vfsmount structure contains a pointer to the root directory entry of the filesystem, of type dentry, in its mnt_root member. dentry structures are generic structures used to represent files, directories, pipes, sockets, and all other file types within the kernel. They have no 1:1 mapping on disk and there can be multiple dentry structures per inode.

Once the root dentry structure is obtained, we can then recursively traverse the entire filesystem by walking each dentry’s d_u.d_child member. For each entry in the child list, we check if it’s a directory or a file. If it’s a directory, the plugin simply creates the directory in the output directory’s hierarchy and then recursively processes it. If it’s a file, then we need to recover its contents from memory and create the file in the output hierarchy.  For both directories and files, we need to update the metadata of the newly created files to match those in memory. This process effectively recovers the entire filesystem to disk in the exact order and layout as in memory.

3) Recovering metadata

Recovering metadata is a simple process because all the information is directly stored within the inode structure of the file or directory, which is stored in the d_inode member of the dentry structure.  This line of code from the plugin accesses all the metadata from the inode that is currently recovered:

(perms, size, atime, mtime) = (inode.i_mode, inode.i_size, inode.i_atime, inode.i_mtime)

4) Recovering file contents

In order to recover a particular file’s contents, we need to be able to locate the structures that track its memory pages within the kernel’s page cache and then find the actual physical pages that store the contents.

The page cache is a performance enhancing mechanism used by the kernel to cache physical pages in RAM so that they do not have to be read from the raw device or network device on which they are stored each time they are accessed. For example, on an active SSH server, a number of files (passwd, shadow, sshd_config, /bin/bash, etc) are going to be constantly read as users log in and off the system. If the kernel had to fully read in these files from disk to memory each time a user logged in, performance would be unacceptably bad.  To alleviate this, pages that are read in from disk are then put in the page cache, so the next time an applications needs to be executed or reads a file, the kernel can simply return the page from the cache without having to touch the disk. An aside: If you have ever dd’ed a disk image in Linux and watched ‘top’, you may have noticed that your entire RAM contents was being used inside the kernel. Much of that was the kernel reading ahead on the disk being imaged and filling the page cache with the next blocks to be processed by dd.

Since the filesystem type we want to recover (tmpfs) is stored only in memory, it is implemented so that all its pages are always stored within the page cache. This lets the filesystem work as all others do, and the kernel will never look for a disk or other source to read from because the files will always be up-to-date within the cache.  

 In order to locate the pages of a particular file, we need to use its index into the page cache. This is stored within the dentry.d_inode.i_mapping member which is of type address_space.  address_space structures are used to track sets of sparse physical pages that together form a contiguous data unit (in this case, a file within a tmpfs mount).  To access the pages of a file inside its address_space, we need to walk the radix tree of struct pages stored within the address_space’s page_tree member. A struct page tracks a physical page and lets the kernel determine the physical address of the page’s contents.

To find the correct index in the tree for a page, we simply divide the offset of the page into the file by page size (4k in this case). We do this in a loop for the entire file in order to recover each page sequentially.  I will save the reader the pain of having to understand the kernel’s radix tree, but the interested reader can look at the radix_tree_lookup_slot function in the tmpfs plugin.  The page cache and its storage format is also very nicely explained in Understanding the Linux Kernel 3rd Edition. Once we are able to walk the page_tree of a particular file and recover each index (a struct page), we then need to determine the physical address of the page it tracks.

Finding the physical address is performed by indexing the struct page into the mem_map array. This array holds a struct page for every physical page of memory. Indexing into mem_map will give us the page number of the physical page, which we can then shift left by PAGE_SHIFT (12) in order to get the page’s physical offset. This process roughly corresponds to the page_to_pfn macro within the kernel.

Once we have the physical offset, all we have to do is read it using Volatility’s API and then concatenate each page’s contents together into the file. At this point we will have recovered the entire filesystem along with its contents and metadata.

Conclusion

This blog post has showcased a new and exciting Volatility plugin. Being able to fully recover tmpfs filesystems out of memory greatly adds to the depth of Volatility’s Linux support for both traditional Linux installs as well as Android devices.  By incorporating this plugin into your forensics and incident response processes, you will recover information that until now has large been ignored or missing.

Please check back frequently as I will be show casing more plugins leading up to the release of Volatility 2.2 at the OMFW Conference. If you are interested in the latest research in memory forensics, I highly suggest you register for and attend OMFW as many of the best memory forensics researchers will be presenting and attending.

If you have any questions or comments about this post, please Email me, catch me on Twitter (@attrc), or reply in the comments section.