Commit Graph
84 Commits
Author SHA1 Message Date
Mike Auty 9711793503 Ensure intel.read(pad=True) doesn't return InvalidAddressExceptions. 2018-06-04 09:40:26 +01:00
Mike Auty 037ac28534 Move around the scanning functions and create an API for the iterators. 2018-05-14 01:03:43 +01:00
Mike Auty 04d609064a Remove TODO that's in the wrong place, and we've just to-done. 2018-05-13 21:32:05 +01:00
Mike Auty 63a1fb3690 Simplify intel scanning to ensure overlaps are handled correctly. 2018-05-13 20:57:11 +01:00
Mike Auty 8b5c58fee9 Fix up slight intel scanner issue. 2018-05-13 15:46:02 +01:00
Mike Auty 5c0140ed74 Fix typo in intel scanning engine. 2018-05-10 02:40:08 +01:00
Mike Auty 83eb346ba5 Clarify logic and fix an issue in intel scanning. 2018-05-10 02:13:46 +01:00
Mike Auty 19b8a0f696 Improve scan chunking for intel layers
We now use mapping to skip unmapped pages when scanning.  This
should be slightly quicker and likely more accurate.  We also now
don't chunk over the size requested by the scanner.  We don't quite
handle overlaps correctly yet (we overlap, but don't prevent
duplicates if the pattern is entirely within the overlap segment).
2018-05-09 23:32:58 +01:00
Mike Auty b791b6c1a1 Add an ignore_errors check for positive lengths. 2018-05-09 01:16:47 +01:00
Mike Auty 983eabac86 Fix an issue with IntelPAE's mapping domain size. 2018-05-08 00:02:14 +01:00
Mike Auty e53cb16485 Vastly speed-up intel memory traversals
This uses the exceptions to indicate how much can be skipped in the
virtual intel translation layer.  This means large holes in the memory
can be skipped more quickly.  It also eliminates the is_valid/mapping
loop.
2018-05-07 23:10:01 +01:00
Mike Auty 9512cbe9eb Commit metadata changeset.
Layers now accept metadata dictionaries (and chain/stack them on top of
those from lower layers).  Metadata can only be set at construction
time, and the metadata dictionary is readonly.  The hope is this will
make enumerating metadata keys across the codebase simpler.

The current metadata items that layers hold is:

architecture (Unknown | Intel32 | Intel64)
os (Unknown | Windows | Linux)
pae (bool)
page_map_offset (int)

This patchset may develop further to help enumerate all of these
(through a registration/reporting system).
2018-04-26 12:48:14 +01:00
Mike Auty 470c139e81 Throw swap exceptions if the page is supposed to be swapped out, but the swap layer can't be found. 2018-03-23 00:48:28 +00:00
Mike Auty 3f37ab760c Change inheritance to fix type checking. 2018-03-08 00:23:49 +00:00
Mike Auty e8422f0774 Refactor to reduce duplication in swap support. 2018-03-05 23:53:34 +00:00
Mike Auty 994bd7c394 Fix up the finding of swap layers so swap actually works. 2018-02-11 21:41:39 +00:00
Mike Auty b7c68f4e98 Rework LayerListRequirement from a ListRequirement to a MultiRequirement. 2018-02-11 18:35:18 +00:00
Mike Auty cd8f6876ff Add in LayerListRequirement type. 2018-02-10 22:42:59 +00:00
Mike Auty 24be045266 Minor fix ups with typing and exceptions. 2018-02-10 22:42:31 +00:00
Mike Auty cbe75dadf7 Add in Intel32e and improve IntelPAE. 2018-02-10 00:03:13 +00:00
Mike Auty 8b75c458ba Fix up the calculations and filtering for pagefile offsets. 2018-02-09 01:19:22 +00:00
Mike Auty deb7da246b Add in initial shot at PAE page entry handling. 2018-01-28 16:29:28 +00:00
Mike Auty 35be4c5a84 Add in more information to the paged exceptions. 2018-01-28 16:25:16 +00:00
Mike Auty c203663f0b Refactor requirements to support a list of swap layers. 2018-01-28 14:48:50 +00:00
Mike Auty ac12098317 Add type-annotations for reamining layer files. 2017-12-13 20:48:52 +00:00
Mike Auty 30059766b6 Plug the WindowsMixin into the automagic. 2017-11-24 12:02:34 +00:00
Mike Auty d9d67efa3a Add in the capability to stash linux_banners.
Since the stacker and Linux automagic will always be different objects
(even if they're the same class), and we don't want to add OS specific
code to the stacker, this is the simplest way of allowing the Linux
symbol automagic to get information from the dtb finding stacker.
2017-11-04 17:09:59 +00:00
Mike Auty 761777c8ef Add commentary as to why we can't scan faster by ignoring surjective duplicates. 2017-06-03 20:57:19 +01:00
Mike Auty b72c3aaed5 Handle invalid translations lower than the top layer when scanning. 2017-06-03 20:26:21 +01:00
Mike Auty 5e15d522bd Remove automatic IDE-inserted attributions.
They weren't consistent and they show just remind me how slowly
I've been getting vol 3 written!  5:S

There's git commits to track all of that stuff if we need it.
2017-05-14 11:40:49 +01:00
Mike Auty de8508a8bb Reset the maxvirtaddr to use maxphyaddr just so we don't forget. 2017-05-01 14:35:22 +01:00
Mike Auty e182e44815 Remove all hardcoded PAGE_SIZE variables, and expose more Intel variables. 2017-04-30 22:09:26 +01:00
Mike Auty 687da20cbc Fix up a typo @npetroni pointed out. 2017-04-20 23:11:42 +01:00
Mike Auty d6728fc2b4 Add in attributes and appropriate validation of TranslationLayers based on those attributes. 2017-02-10 13:57:44 +00:00
Mike Auty 1437e24b53 Add a note about _maxphyaddr. 2017-01-02 22:17:35 +00:00
Mike Auty bab71d4358 Refactor the progress_callback so it can be used in automagics.
Refactored the location of the progress_callback so that the UI can
specify it and ensure it is used by the automagics process as well as
the plugins themselves.

The progress_callback function has also had an optional description parameter
added, so that it can differentiated in case two scans happen
simultaneously.
2016-12-26 23:45:47 +00:00
Mike Auty a4f27e73f4 Refactor the scanners to allow multithreaded searching. 2016-12-25 17:01:05 +00:00
Mike Auty aad6aa9b4d Rewrite the intel scanning engine (and fix up the internal _translate function to include the mapped layer). 2016-12-21 16:53:52 +00:00
Mike Auty 13b1126b90 Guard against no progress_callback in intel layer. 2016-12-08 16:03:22 +00:00
Mike Auty 183ff67699 Attach the kernel virtual offset to intel spaces. 2016-12-07 16:00:31 +00:00
Mike Auty 20d0e83e1d Ensure we don't ever round down when log-ing. 2016-12-07 00:05:47 +00:00
Mike Auty 734b9a68b8 Ensure we use math.log correctly.
This is unlikely to ever happen (since we use python3)
but still just as well to implement in case something else
similar comes up.

https://www.willmcgugan.com/blog/tech/post/finding-the-first-bit-set-with-python/
2016-12-06 23:40:21 +00:00
Mike Auty a0f7c81128 Reveal the register size in intel spaces. 2016-12-04 02:14:54 +00:00
Mike Auty d2fcc1d4b0 Add in optimization technique to reduce the number of pages that are scanned more than once. 2016-11-19 23:34:08 +00:00
Mike Auty c9abd290ec Remove an unnecessary test from the intel scan loop. 2016-11-19 18:05:41 +00:00
Mike Auty f707c0c4e9 Add in first shot at an 'efficient' virtual memory scanner. Uses page misses to calculate skipping distances. 2016-11-18 17:14:48 +00:00
Mike Auty 096e2f4610 Make the paged exceptions more useful at knowing how big an area is unavailable. 2016-11-18 11:40:48 +00:00
Mike Auty 3dc161f9b4 Rework Translation Layers to tie more closely to configurations
This change is quite signficant, and requires that TranslationLayers
get all additional parameters that they need through their requirements.

These are now automatically enumerated and populated on object
construction based on the requirements, so should not require lots of
repetitive filling out of fields.

It does come with the downside that TranslationLayers can only be
contructed with a context (and appropiate config), but TLs in particular
always require a context (to contain the base layer) and blank configs
can be constructed relatively easily (convenience functions can be added
if necessary).

This allows configuration trees to be built up, and their configs
spliced into an existing config (as if it were being loaded from a
file).

Not all ConstructableRequirements use this method, since SymbolTables
(for example) do not have access to the context or config_path in order
to get to any parameters stored in the context's config.  They therefore
are still passed their requirement values as __init__ parameters
instead.
2016-08-22 02:20:10 +01:00
Mike Auty d3e63fbc1a Rename update_configuration to build_configuration and return a standalone config. 2016-08-22 02:18:23 +01:00
Mike Auty ca61cf10b9 Refactor the context back into the ConfigurableInterface.
This also adds support for manually constructed configurables to
populate the config tree in the current context.

I'm still toying around with this though, I need to figure out
what to do with optional values and think the whole thing through
to make sure it's worthwhile.
2016-08-14 21:28:43 +01:00