This simplifies the design of these plugins by moving as much MFTEntry
specific data into the extension class (caching attributes, since
they'll need to be accessed repeatedly) and moving away from the
callback-based implementation to one where classmethods consume
`mft.MFTEntry` objects in order to produce their values.
These changes do two important things:
- They allow us to preserve `object.String` objects until the generator
function, which makes the public interface much better since people
can navigate back the the source of the data within their context
- Completely eliminates the `record_map` that was causing so much memory
consumption.
There was a subtle issue that was causing substantial performance issues
in the MFTScan plugins. The `record_map` was purportedly of type
`Dict[str, Tuple[int, str, int]]`, but in reality, the second member was
a list, and its `str` item was actually being populated with unprocessed
values from method calls on the MFT extension classes, which actually
return `object.String`. These objects are substantially larger than
basic `str` types:
```
[ins] In [5]: pympler.asizeof.asizeof(rec_name)
Out[5]: 312648
[ins] In [6]: pympler.asizeof.asizeof(str(rec_name))
Out[6]: 64
```
This caused this dictionary to grow in size to several gigabytes on
larger samples, resulting in thrashing and OOM errors.
The new layer data type renders the output a little differently, and the
plugin also seems to render 'N/A' for a missing value where previously
it was an empty string.
This fixes all import from statements in the codebase that were
importing things other than modules into module namespaces from other
volatility3 modules. This should prevent accidental re-exporting.
This checks `ast.ImportFrom` statements to see if anything other than
modules are being imported in this way. It enumerates all instances of
this and suggests a fix.
This change sets the `script`, `script-only`, and `primary` requirements
to only apply to the `generic.Volshell` class. `regex-scanner` is okay
to be shared between the base and inherited classes, but `script` and
`script-only` have to be generic-only in order to avoid conflicts when
populating the argparse parser.
`primary` must be generic-only in order to avoid ending up unsatisfied
when superclass requirements require a module, suppressing construction
of the `primary` layer.
Instead of using the tree-sitter third party library, this uses Python's
`ast` module to parse the source code and traverse the tree with a
visitor pattern. This is preferred because it's native to the language
itself, and Python developers are more likely to be familiar with it.
The traversal also handles nested scopes better than the prior
implementation. For example, classes that are declared inside of other
classes can now be looked up even though they don't exist at the top
level of the module namespace, since any time a class definition is
entered, that class is pushed to the top of a stack that can be examined
when visiting inner classes.
This also adds lots of log messages at different levels, plus a command
line argument for specifying verbosity, which should help with debugging
down the line.
David McDonald