Merge branch 'volatilityfoundation:develop' into fix/ssl-error-for-python37

This commit is contained in:
Donghyun Kim
2022-11-08 10:39:47 +09:00
committed by GitHub
11 changed files with 372 additions and 17 deletions
+3 -1
View File
@@ -111,9 +111,11 @@ needs_sphinx = '2.0'
# ones.
extensions = [
'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.napoleon', 'sphinx.ext.intersphinx', 'sphinx.ext.todo',
'sphinx.ext.coverage', 'sphinx.ext.viewcode'
'sphinx.ext.coverage', 'sphinx.ext.viewcode', 'sphinx.ext.autosectionlabel'
]
autosectionlabel_prefix_document = True
try:
import sphinx_autodoc_typehints
@@ -0,0 +1,192 @@
Linux Tutorial
==============
This guide will give you a brief overview of how volatility3 works as well as a demonstration of several of the plugins available in the suite.
Acquiring memory
----------------
Volatility3 does not provide the ability to acquire memory. Below are some examples of tools that can be used to acquire memory, but more are available:
* `AVML - Acquire Volatile Memory for Linux <https://github.com/microsoft/avml>`_
* `LiME - Linux Memory Extract <https://github.com/504ensicsLabs/LiME>`_
Procedure to create symbol tables for linux
--------------------------------------------
To create a symbol table please refer to :ref:`symbol-tables:Mac or Linux symbol tables`.
.. tip:: It may be possible to locate pre-made ISF files from the `Linux ISF Server <https://isf-server.techanarchy.net/>`_ ,
which is built and maintained by `kevthehermit <https://twitter.com/kevthehermit>`_.
After creating the file or downloading it from the ISF server, place the file under the directory ``volatility3/symbols/linux``.
If necessary create a linux directory under the symbols directory (this will become unnecessary in future versions).
Listing plugins
---------------
The following is a sample of the linux plugins available for volatility3, it is not complete and more more plugins may
be added. For a complete reference, please see the volatility 3 :doc:`list of plugins <volatility3.plugins>`.
For plugin requests, please create an issue with a description of the requested plugin.
.. code-block:: shell-session
$ python3 vol.py --help | grep -i linux. | head -n 5
banners.Banners Attempts to identify potential linux banners in an
linux.bash.Bash Recovers bash command history from memory.
linux.check_afinfo.Check_afinfo
linux.check_creds.Check_creds
linux.check_idt.Check_idt
.. note:: Here the the command is piped to grep and head in-order to provide the start of the list of linux plugins.
Using plugins
-------------
The following is the syntax to run the volatility CLI.
.. code-block:: shell-session
$ python3 vol.py -f <path to memory image> <plugin_name> <plugin_option>
Example
-------
banners
~~~~~~~
In this example we will be using a memory dump from the Insomni'hack teaser 2020 CTF Challenge called Getdents. We will limit the discussion to memory forensics with volatility 3 and not extend it to other parts of the challenge.
Thanks go to `stuxnet <https://github.com/stuxnet999/>`_ for providing this memory dump and `writeup <https://stuxnet999.github.io/insomnihack/2020/09/17/Insomihack-getdents.html>`_.
.. code-block:: shell-session
$ python3 vol.py -f memory.vmem banners
Volatility 3 Framework 2.0.1
Progress: 100.00 PDB scanning finished
Offset Banner
0x141c1390 Linux version 4.15.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 (Ubuntu 4.15.0-42.45-generic 4.15.18)
0x63a00160 Linux version 4.15.0-72-generic (buildd@lcy01-amd64-026) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 (Ubuntu 4.15.0-72.81-generic 4.15.18)
0x6455c4d4 Linux version 4.15.0-72-generic (buildd@lcy01-amd64-026) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 (Ubuntu 4.15.0-72.81-generic 4.15.18)
0x6e1e055f Linux version 4.15.0-72-generic (buildd@lcy01-amd64-026) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 (Ubuntu 4.15.0-72.81-generic 4.15.18)
0x7fde0010 Linux version 4.15.0-72-generic (buildd@lcy01-amd64-026) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 (Ubuntu 4.15.0-72.81-generic 4.15.18)
The above command helps us to find the memory dump's kernel version and the distribution version. Now using the above banner we can search for the needed ISF file from the ISF server.
If ISF file cannot be found then, follow the instructions on :ref:`getting-started-linux-tutorial:Procedure to create symbol tables for linux`. After that, place the ISF file under the ``volatility3/symbols/linux`` directory.
.. tip:: Use the banner text which is most repeated to search from ISF Server.
linux.pslist
~~~~~~~~~~~~
.. code-block:: shell-session
$ python3 vol.py -f memory.vmem linux.pslist
Volatility 3 Framework 2.0.1 Stacking attempts finished
PID PPID COMM
1 0 systemd
2 0 kthreadd
3 2 kworker/0:0
4 2 kworker/0:0H
5 2 kworker/u256:0
6 2 mm_percpu_wq
7 2 ksoftirqd/0
8 2 rcu_sched
9 2 rcu_bh
10 2 migration/0
11 2 watchdog/0
12 2 cpuhp/0
13 2 kdevtmpfs
14 2 netns
15 2 rcu_tasks_kthre
16 2 kauditd
.....
``linux.pslist`` helps us to list the processes which are running, their PIDs and PPIDs.
linux.pstree
~~~~~~~~~~~~
.. code-block:: shell-session
$ python3 vol.py -f memory.vmem linux.pstree
Volatility 3 Framework 2.0.1
Progress: 100.00 Stacking attempts finished
PID PPID COMM
1 0 systemd
* 636 1 polkitd
* 514 1 acpid
* 1411 1 pulseaudio
* 517 1 rsyslogd
* 637 1 cups-browsed
* 903 1 whoopsie
* 522 1 ModemManager
* 525 1 cron
* 526 1 avahi-daemon
** 542 526 avahi-daemon
* 657 1 unattended-upgr
* 914 1 kerneloops
* 532 1 dbus-daemon
* 1429 1 ibus-x11
* 929 1 kerneloops
* 1572 1 gsd-printer
* 933 1 upowerd
* 1071 1 rtkit-daemon
* 692 1 gdm3
** 1234 692 gdm-session-wor
*** 1255 1234 gdm-x-session
**** 1257 1255 Xorg
**** 1266 1255 gnome-session-b
***** 1537 1266 gsd-clipboard
***** 1539 1266 gsd-color
***** 1542 1266 gsd-datetime
***** 2950 1266 deja-dup-monito
***** 1546 1266 gsd-housekeepin
***** 1548 1266 gsd-keyboard
***** 1550 1266 gsd-media-keys
``linux.pstree`` helps us to display the parent child relationships between processes.
linux.bash
~~~~~~~~~~
Now to find the commands that were run in the bash shell by using ``linux.bash``.
.. code-block:: shell-session
$ python3 vol.py -f memory.vmem linux.bash
Volatility 3 Framework 2.0.1
Progress: 100.00 Stacking attempts finished
PID Process CommandTime Command
1733 bash 2020-01-16 14:00:36.000000 sudo reboot
1733 bash 2020-01-16 14:00:36.000000 AWAVH
1733 bash 2020-01-16 14:00:36.000000 sudo apt upgrade
1733 bash 2020-01-16 14:00:36.000000 sudo apt upgrade
1733 bash 2020-01-16 14:00:36.000000 sudo reboot
1733 bash 2020-01-16 14:00:36.000000 sudo apt update
1733 bash 2020-01-16 14:00:36.000000 sudo apt update
1733 bash 2020-01-16 14:00:36.000000 sudo reboot
1733 bash 2020-01-16 14:00:36.000000 sudo apt upgrade
1733 bash 2020-01-16 14:00:36.000000 sudo apt update
1733 bash 2020-01-16 14:00:36.000000 rub
1733 bash 2020-01-16 14:00:36.000000 sudo apt upgrade
1733 bash 2020-01-16 14:00:36.000000 uname -a
1733 bash 2020-01-16 14:00:36.000000 uname -a
1733 bash 2020-01-16 14:00:36.000000 sudo apt autoclean
1733 bash 2020-01-16 14:00:36.000000 sudo reboot
1733 bash 2020-01-16 14:00:36.000000 sudo apt upgrade
1733 bash 2020-01-16 14:00:41.000000 chmod +x meterpreter
1733 bash 2020-01-16 14:00:42.000000 sudo ./meterpreter
@@ -0,0 +1,124 @@
Windows Tutorial
================
This guide provides a brief introduction to how volatility3 works as a demonstration of several of the plugins available in the suite.
Acquiring memory
----------------
Volatility does not provide the ability to acquire memory.
Memory can be acquired using a number of tools, below are some examples but others exist:
* `WinPmem <https://github.com/Velocidex/WinPmem/releases/latest>`_
* `FTK Imager <https://accessdata.com/product-download/ftk-imager-version-4-5>`_
Listing Plugins
---------------
The following is a sample of the windows plugins available for volatility3, it is not complete and more more plugins may
be added. For a complete reference, please see the volatility 3 :doc:`list of plugins <volatility3.plugins>`.
For plugin requests, please create an issue with a description of the requested plugin.
.. code-block:: shell-session
$ python3 vol.py --help | grep windows | head -n 5
windows.bigpools.BigPools
windows.cmdline.CmdLine
windows.crashinfo.Crashinfo
windows.dlllist.DllList
.. note:: Here the the command is piped to grep and head in-order to provide the start of a list of the available windows plugins.
Using plugins
-------------
The following is the syntax to run the volatility CLI.
.. code-block:: shell-session
$ python3 vol.py -f <path to memory image> plugin_name plugin_option
Example
-------
windows.pslist
~~~~~~~~~~~~~~
In this example we will be using a memory dump from the PragyanCTF'22.
We will limit the discussion to memory forensics with volatility 3 and not extend it to other parts of the challenges.
When using windows plugins in volatility 3, the required ISF file can often be generated from PDB files automatically
downloaded from Microsoft servers, and therefore does not require locating or adding specific ISF files to the volatility 3 symbols directory.
.. code-block:: shell-session
$ python3 vol.py -f MemDump.DMP windows.pslist | head -n 10
Volatility 3 Framework 2.0.1 PDB scanning finished
PID PPID ImageFileName Offset(V) Threads Handles SessionId Wow64 CreateTime ExitTime File output
4 0 System 0xfa8000cbc040 85 492 N/A False 2022-02-07 16:30:12.000000 N/A Disabled
276 4 smss.exe 0xfa8001e04040 2 29 N/A False 2022-02-07 16:30:12.000000 N/A Disabled
352 336 csrss.exe 0xfa8002110b30 9 375 0 False 2022-02-07 16:30:13.000000 N/A Disabled
404 336 wininit.exe 0xfa800219f060 3 74 0 False 2022-02-07 16:30:13.000000 N/A Disabled
412 396 csrss.exe 0xfa80021c5b30 9 224 1 False 2022-02-07 16:30:13.000000 N/A Disabled
468 396 winlogon.exe 0xfa8002284060 5 113 1 False 2022-02-07 16:30:14.000000 N/A Disabled
``windows.pslist`` helps list the processes running while the memory dump was taken.
windows.pstree
~~~~~~~~~~~~~~
.. code-block:: shell-session
$ python3 vol.py -f MemDump.DMP windows.pstree | head -n 20
Volatility 3 Framework 2.0.1 PDB scanning finished
PID PPID ImageFileName Offset(V) Threads Handles SessionId Wow64 CreateTime ExitTime
4 0 System 0xfa8000cbc040 85 492 N/A False 2022-02-07 16:30:12.000000 N/A
* 276 4 smss.exe 0xfa8001e04040 2 29 N/A False 2022-02-07 16:30:12.000000 N/A
352 336 csrss.exe 0xfa8002110b30 9 375 0 False 2022-02-07 16:30:13.000000 N/A
404 336 wininit.exe 0xfa800219f060 3 74 0 False 2022-02-07 16:30:13.000000 N/A
* 504 404 services.exe 0xfa80022ccb30 7 190 0 False 2022-02-07 16:30:14.000000 N/A
** 960 504 svchost.exe 0xfa8001c17b30 39 1003 0 False 2022-02-07 16:30:14.000000 N/A
** 1216 504 svchost.exe 0xfa80026e0b30 18 311 0 False 2022-02-07 16:30:15.000000 N/A
** 1312 504 svchost.exe 0xfa8002740380 19 287 0 False 2022-02-07 16:30:15.000000 N/A
** 1984 504 taskhost.exe 0xfa8002eb1b30 8 129 1 False 2022-02-07 16:30:27.000000 N/A
** 804 504 svchost.exe 0xfa80024ca5f0 20 450 0 False 2022-02-07 16:30:14.000000 N/A
*** 100 804 audiodg.exe 0xfa80025b4b30 6 131 0 False 2022-02-07 16:30:14.000000 N/A
** 1568 504 SearchIndexer. 0xfa800254b480 12 616 0 False 2022-02-07 16:30:32.000000 N/A
** 744 504 svchost.exe 0xfa8002477b30 8 265 0 False 2022-02-07 16:30:14.000000 N/A
** 1096 504 svchost.exe 0xfa800260db30 14 357 0 False 2022-02-07 16:30:14.000000 N/A
** 616 504 svchost.exe 0xfa8002b86ab0 13 314 0 False 2022-02-07 16:32:16.000000 N/A
** 624 504 svchost.exe 0xfa8002410630 10 350 0 False 2022-02-07 16:30:14.000000 N/A
``windows.pstree`` helps to display the parent child relationships between processes.
.. note:: Here the the command is piped to head in-order to provide smaller output, here listing only the first 20.
windows.hashdump
~~~~~~~~~~~~~~~~
.. code-block:: shell-session
$ python3 vol.py -f MemDump.DMP windows.hashdump
Volatility 3 Framework 2.0.3
Progress: 100.00 PDB scanning finished
User rid lmhash nthash
Administrator 500 aad3b435b51404eeaad3b435b51404ee 31d6cfe0d16ae931b73c59d7e0c089c0
Guest 501 aad3b435b51404eeaad3b435b51404ee 31d6cfe0d16ae931b73c59d7e0c089c0
Frank Reynolds 1000 aad3b435b51404eeaad3b435b51404ee a88d1e18706d3aa676e01e5943d15911
HomeGroupUser$ 1002 aad3b435b51404eeaad3b435b51404ee af10ecac6ea817d2bb56e3e5c33ce1cd
Dennis 1003 aad3b435b51404eeaad3b435b51404ee cf96684bbc7877920adaa9663698bf54
``windows.hashdump`` helps to list the hashes of the users in the system.
+13 -3
View File
@@ -7,9 +7,10 @@ Volatility 3 is Open Source.
:doc:`List of plugins <volatility3.plugins>`
Here are some guidelines for using Volatility 3 effectively:
Below is the main documentation regarding volatility 3:
.. toctree::
:caption: Documentation
basics
development
@@ -18,12 +19,21 @@ Here are some guidelines for using Volatility 3 effectively:
volshell
glossary
Python Packages
===============
There is also some information to get you started quickly:
.. toctree::
:caption: Getting Started
getting-started-linux-tutorial
getting-started-windows-tutorial
.. toctree::
:caption: Python Packages
volatility3
Indices and tables
==================
+2 -2
View File
@@ -40,8 +40,8 @@ following command:
The :envvar:`PYTHONPATH` environment variable is not required if the Volatility library is installed in the system's library path
or a virtual environment.
Mac/Linux symbol tables
-----------------------
Mac or Linux symbol tables
--------------------------
For Mac/Linux systems, both use the same mechanism for identification. The generated files contain an identifying string (the operating system
banner), which Volatility's automagic can detect. Volatility caches the mapping between the strings and the symbol
+1 -1
View File
@@ -40,7 +40,7 @@ setuptools.setup(name = "volatility3",
'': ['development', 'development.*'],
'development': ['*']
},
packages = setuptools.find_packages(exclude = ["development", "development.*"]),
packages = setuptools.find_namespace_packages(exclude = ["development", "development.*"]),
entry_points = {
'console_scripts': [
'vol = volatility3.cli:main',
@@ -2,6 +2,7 @@
# which is available at https://www.volatilityfoundation.org/license/vsl-v1.0
#
import base64
import datetime
import json
import logging
import os
@@ -157,10 +158,10 @@ class SqliteCache(CacheManagerInterface):
_required_framework_version = (2, 0, 0)
_version = (1, 0, 0)
cache_period = '-3 days'
def __init__(self, filename: str):
super().__init__(filename)
self.cache_period = constants.SQLITE_CACHE_PERIOD
try:
self._database = self._connect_storage(filename)
except sqlite3.DatabaseError:
@@ -170,6 +171,7 @@ class SqliteCache(CacheManagerInterface):
def _connect_storage(self, path: str) -> sqlite3.Connection:
database = sqlite3.connect(path)
database.row_factory = sqlite3.Row
database.cursor().execute(
f'CREATE TABLE IF NOT EXISTS database_info (schema_version INT DEFAULT {constants.CACHE_SQLITE_SCHEMA_VERSION})')
schema_version = database.cursor().execute('SELECT schema_version FROM database_info').fetchone()
@@ -259,10 +261,31 @@ class SqliteCache(CacheManagerInterface):
cache_update = set()
files_to_timestamp = on_disk_locations.intersection(cached_locations)
if files_to_timestamp:
result = self._database.cursor().execute("SELECT location FROM cache WHERE local = 1 "
result = self._database.cursor().execute("SELECT location, cached FROM cache WHERE local = 1 "
f"AND cached < date('now', '{self.cache_period}');")
for row in result:
if row['location'] in files_to_timestamp:
location = row['location']
stored_timestamp = datetime.datetime.fromisoformat(row['cached'])
timestamp = stored_timestamp # Default to requiring update
# See if the file is a local URL type we can handle:
parsed = urllib.parse.urlparse(location)
pathname = None
if parsed.scheme == 'file':
pathname = urllib.request.url2pathname(parsed.path)
if parsed.scheme == 'jar':
inner_url = urllib.parse.urlparse(parsed.path)
if inner_url.scheme == 'file':
pathname = inner_url.path.split('!')[0]
if pathname:
timestamp = datetime.datetime.fromtimestamp(os.stat(pathname).st_mtime)
else:
vollog.log(constants.LOGLEVEL_VVVV,
"File location in database classed as local but not file/jar URL")
# If we're supposed to include it, and our last check is older than (or equal to) the file timestamp
if row['location'] in files_to_timestamp and stored_timestamp < timestamp:
cache_update.add(row['location'])
idextractors = list(framework.class_subclasses(IdentifierProcessor))
+4 -1
View File
@@ -40,7 +40,7 @@ BANG = "!"
# We use the SemVer 2.0.0 versioning scheme
VERSION_MAJOR = 2 # Number of releases of the library with a breaking change
VERSION_MINOR = 4 # Number of changes that only add to the interface
VERSION_PATCH = 0 # Number of changes that do not change the interface
VERSION_PATCH = 1 # Number of changes that do not change the interface
VERSION_SUFFIX = ""
# TODO: At version 2.0.0, remove the symbol_shift feature
@@ -63,6 +63,9 @@ LOGLEVEL_VVVV = 6
CACHE_PATH = os.path.join(os.path.expanduser("~"), ".cache", "volatility3")
"""Default path to store cached data"""
SQLITE_CACHE_PERIOD = '-3 days'
"""SQLite time modifier for how long each item is valid in the cache for"""
if sys.platform == 'win32':
CACHE_PATH = os.path.realpath(os.path.join(os.environ.get("APPDATA", os.path.expanduser("~")), "volatility3"))
os.makedirs(CACHE_PATH, exist_ok = True)
+1 -1
View File
@@ -9,7 +9,7 @@ import struct
from typing import Any, ClassVar, Dict, Iterable, List, Optional, Tuple, Type, Union as TUnion, overload
from volatility3.framework import constants, interfaces
from volatility3.framework.objects import templates, utility
from volatility3.framework.objects import templates
vollog = logging.getLogger(__name__)
@@ -323,7 +323,7 @@ class Handles(interfaces.plugins.PluginInterface):
obj_name = item.file_name_with_device()
elif obj_type == "Process":
item = entry.Body.cast("_EPROCESS")
obj_name = f"{utility.array_to_string(proc.ImageFileName)} Pid {item.UniqueProcessId}"
obj_name = f"{utility.array_to_string(item.ImageFileName)} Pid {item.UniqueProcessId}"
elif obj_type == "Thread":
item = entry.Body.cast("_ETHREAD")
obj_name = f"Tid {item.Cid.UniqueThread} Pid {item.Cid.UniqueProcess}"
@@ -28,8 +28,11 @@ class proc(generic.GenericIntelProcess):
if not isinstance(parent_layer, interfaces.layers.TranslationLayerInterface):
raise TypeError("Parent layer is not a translation layer, unable to construct process layer")
with contextlib.suppress(exceptions.InvalidAddressException):
try:
dtb = self.get_task().map.pmap.pm_cr3
except exceptions.InvalidAddressException:
# Bail out because we couldn't find the DTB
return None
if preferred_name is None:
preferred_name = self.vol.layer_name + f"_Process{self.p_pid}"
@@ -38,10 +41,8 @@ class proc(generic.GenericIntelProcess):
return self._add_process_layer(self._context, dtb, config_prefix, preferred_name)
def get_map_iter(self) -> Iterable[interfaces.objects.ObjectInterface]:
with contextlib.suppress(exceptions.InvalidAddressException):
task = self.get_task()
try:
task = self.get_task()
current_map = task.map.hdr.links.next
except exceptions.InvalidAddressException:
return