Instead of constructing a dict and yielding from `.values()` directly,
yields values as they are yielded from the underlying call to
`get_kset_modules`.
Returning a `dict` leads to an all-or-nothing result, where you either
get the complete set of modules, or an exception is encountered and
raised and you get nothing. Using an iterator allows for as many
readable modules to be yielded as possible, and the caller can still
catch any `InvalidAddressException` that gets thrown during iteration.
Fixes regression introduced in #1632
Symbolic links are allocated in the paged pools, not non-paged. This was
causing us to miss symlinks across both pre and post win8 samples.
Removes casts to `int` performed before checking set membership, since
the computed `__hash__` value will be the same for both the Python
primitive and the volatility `objects.Pointer`.
The traversal of `ActiveProcessLinks` from `PsActiveProcessHead` was
only being done in the forward direction; if for some reason
`PsActiveProcessHead` hasn't been updated to point at the 'current' list
head, entries in the backwards traversal direction will be missed.
Adds needed return statements in `exec_flag` method in order to avoid
`InvalidAddressException` when reading from invalid memory after an
`is_valid` check has already been performed.
This fixes a bug in the plugin logic that causes valid entries to be
excluded in the following scenario:
- An entry is discovered but deemed invalid due to unreadable
size/timestamps.
- The offset gets placed into the `seen` tracking set anyway
- Another entry (this time, with valid filesize/timestamps) with the
same physical offset is encountered, but is skipped because this
offset is already in the `seen` tracking set.
This updates the logic to only add the offset to the tracker if the
shimcache entry is valid.
I noticed that many plugins were creating duplicate per-process
translation layers - for instance, `windows.envars.Envars` was ending up
with > 30 process layers per process due to repeated calls to
`get_peb()`, which calls `add_process_space()` internally.
This adds `@functools.lru_cache` to the `get_peb()` and `get_peb32()`
methods on the `EPROCESS` extension, since these should only need to be
created once.
Also adds `@functools.lru_cache` to `add_process_space` to enable
reusing the same process address space, provided the same arguments are
passed to the `add_process_space()` method.
Updates the exception message to report the correct dependency instead
of the layer name itself. Instead of reporting the actual dependency, it
was reporting, for example 'Layer layer_name is depended upon by
layer_name'.
This exclusion of threads where there are < 5 vads seems to filter valid
threads (at least, threads where the start address or Win32 start
address values are readable and valid disassembly, and the start time
makes sense in the context of the parent process).
Tracebacks were occurring across a number of samples when running the
threads/threadscan plugins due to uncaught `InvalidAddressExceptions`.
Further investigations led to the discovery of some incorrect thread
filtering that was missing valid threads.
This improves type-hinting in the `ThrdScan` class and in the `ETHREAD`
extension class through narrowing the return type of some methods from
`interfaces.objects.ObjectInterface` to their actual return type,
`extensions.ETHREAD`.
Also creates a `NamedTuple` for holding thread info, which cleans up the
type signature and makes the returned value easier for consumers to use.
Previously, the InodePages plugin wasn't rendering treegrid columns when
the `--dump` flag was passed. This fixes that, and adds an additional
`Output File` column that displays the name of the file containing the
dumped data.
A `SymbolError` can occur when a type contains a pointer to an opaque
type. For example, `_EPROCESS` can have a member that points to an
`_EPROCESS_QUOTA_BLOCK`, but there is no definition for that type, so
its size and readability can't be determined.
This wraps the block in a try/except, and reports that the type has an
unknown size in the suffix if a `SymbolError` occurs.
Improves the naming of a couple of the new extension class methods to
more accurately reflect the return type, and adds docstrings to
extensions class methods.
Updates type hints on some fields of the result namedtuples to be their
`objects.Primitive` types instead of Python primitives, and does any
conversion to Python primitives in the generator methods.
This simplifies the design of these plugins by moving as much MFTEntry
specific data into the extension class (caching attributes, since
they'll need to be accessed repeatedly) and moving away from the
callback-based implementation to one where classmethods consume
`mft.MFTEntry` objects in order to produce their values.
These changes do two important things:
- They allow us to preserve `object.String` objects until the generator
function, which makes the public interface much better since people
can navigate back the the source of the data within their context
- Completely eliminates the `record_map` that was causing so much memory
consumption.
There was a subtle issue that was causing substantial performance issues
in the MFTScan plugins. The `record_map` was purportedly of type
`Dict[str, Tuple[int, str, int]]`, but in reality, the second member was
a list, and its `str` item was actually being populated with unprocessed
values from method calls on the MFT extension classes, which actually
return `object.String`. These objects are substantially larger than
basic `str` types:
```
[ins] In [5]: pympler.asizeof.asizeof(rec_name)
Out[5]: 312648
[ins] In [6]: pympler.asizeof.asizeof(str(rec_name))
Out[6]: 64
```
This caused this dictionary to grow in size to several gigabytes on
larger samples, resulting in thrashing and OOM errors.
The new layer data type renders the output a little differently, and the
plugin also seems to render 'N/A' for a missing value where previously
it was an empty string.
This fixes all import from statements in the codebase that were
importing things other than modules into module namespaces from other
volatility3 modules. This should prevent accidental re-exporting.
This checks `ast.ImportFrom` statements to see if anything other than
modules are being imported in this way. It enumerates all instances of
this and suggests a fix.
This change sets the `script`, `script-only`, and `primary` requirements
to only apply to the `generic.Volshell` class. `regex-scanner` is okay
to be shared between the base and inherited classes, but `script` and
`script-only` have to be generic-only in order to avoid conflicts when
populating the argparse parser.
`primary` must be generic-only in order to avoid ending up unsatisfied
when superclass requirements require a module, suppressing construction
of the `primary` layer.
Instead of using the tree-sitter third party library, this uses Python's
`ast` module to parse the source code and traverse the tree with a
visitor pattern. This is preferred because it's native to the language
itself, and Python developers are more likely to be familiar with it.
The traversal also handles nested scopes better than the prior
implementation. For example, classes that are declared inside of other
classes can now be looked up even though they don't exist at the top
level of the module namespace, since any time a class definition is
entered, that class is pushed to the top of a stack that can be examined
when visiting inner classes.
This also adds lots of log messages at different levels, plus a command
line argument for specifying verbosity, which should help with debugging
down the line.
This adds a script and GitHub action to the `test` directory that
dynamically imports all modules in `volatility3`, searches for usages of
`VersionableInterface` objects within classes that inherit from
`ConfigurableInterface` but don't enumerate the used component as a
requirement in `get_requirements()`, and returns -1 if any violations
are found.
Fixes
When I updated the requirements for `Consoles` in #1738, I bumped the
version number on the `VerInfo` requirement instead of on the `Info`
requirement.
closes#1741
This adds `interfaces.configuration.VersionableInterface` as a
superclass to `TimelinerInterface` in order to be consistent with other
versioned interfaces such as `PluginInterface`.
Per code review recommendations, splits the `_unmarshall` classmethod
into two components, one of which retrieves the raw value, and the other
that returns the masked pointer. The `get_raw_value` method now calls
the `_get_raw_value` classmethod using its instance information.
Made an additive change to `Pointer` by adding the `get_raw_value()`
method, so bumping the minor version here. The `get_raw_dpc()` method
was removed from the `KTIMER` extension class, which is currently
unversioned.
This removes the `get_raw_dpc` method from the `KTIMER` extension class.
This method was inaccurate in that it actually returns the masked
pointer value instead of the full 64-bit value encoded in that member,
which is required in order to correctly decode the 'real' pointer.
The invocation of `get_raw_dpc()` was replaced with
`self.Dpc.get_raw_value()`, which was added in the previous commit.
This creates a `get_raw_value()` method for the `Pointer` class that
allows users to access the raw (unmasked) value of a pointer. This was
required in order to decode the encoded `Dpc` pointer that is part of
the `_KTIMER` Windows type. Addition of this type was favored over a
cast to `unsigned long` or `unsigned long long` due to the potential for
future instability of this type due to compiler changes.
See https://github.com/volatilityfoundation/volatility3/issues/1041 for
further discussion around the conversion of `log unsigned int` to
`unsigned long` in `clang`.
See https://github.com/volatilityfoundation/volatility3/pull/1177#discussion_r1650049299
for the original discussion around how to access this pointer in the
`Timers` plugin.
The original code was still returning this as a pointer that ended up
dereferenced in later steps. However, this pointer value actually needs
to be cast to an `unsigned long long` and decoded first.
Something happened when picking _EPROCESS members before that caused
this to not function properly. This one relies on a more stable type
removal instead of _EPROCESS members.
This was originally done to solve a problem where `f32` wasn't available
in the native types from every kernel version. However, it ended up not
being necessary - we can just omit `native_types` from the method call,
and it will construct the types as-needed using the definition for
`float` in `base_types` from the JSON files.
David McDonald