Mike Auty
c6724a538a
Initial support for compressed files.
2017-08-11 01:12:50 +01:00
Mike Auty
807b5dcaf0
Ensure we throw an appropriate error if there's no patterns available.
2017-07-16 22:52:56 +01:00
Mike Auty
c4fbaf248e
Max the number of hash buckets needed precise, and explicitly linked to the hash function.
2017-07-16 01:27:20 +01:00
Mike Auty
f9375d0797
Reduce pointless memory usage, and remove old imports and debugging exception handlers.
2017-07-16 01:23:31 +01:00
Mike Auty
0ca78bc890
Add an efficient multi-string search module (based on Wu-Manber 1994).
...
Empirically this seems far quicker than either Aho-Corasick or
Commentz-Walter for our dataset (very many long search terms).
2017-07-16 01:13:49 +01:00
Mike Auty
2b718f104d
Remove imports that will break the code.
2017-07-14 01:15:52 +01:00
Mike Auty
dd7c9d777a
Refactor the scanners to allow for better organization.
2017-07-13 01:30:01 +01:00
Mike Auty
761777c8ef
Add commentary as to why we can't scan faster by ignoring surjective duplicates.
2017-06-03 20:57:19 +01:00
Mike Auty
c05599d78a
Minor clean-up of segmented layer.
2017-06-03 20:26:45 +01:00
Mike Auty
b72c3aaed5
Handle invalid translations lower than the top layer when scanning.
2017-06-03 20:26:21 +01:00
Mike Auty
5e15d522bd
Remove automatic IDE-inserted attributions.
...
They weren't consistent and they show just remind me how slowly
I've been getting vol 3 written! 5:S
There's git commits to track all of that stuff if we need it.
2017-05-14 11:40:49 +01:00
Mike Auty
6da8076050
Add in Regular Expression scanner.
2017-05-06 14:02:09 +01:00
Mike Auty
de8508a8bb
Reset the maxvirtaddr to use maxphyaddr just so we don't forget.
2017-05-01 14:35:22 +01:00
Mike Auty
e182e44815
Remove all hardcoded PAGE_SIZE variables, and expose more Intel variables.
2017-04-30 22:09:26 +01:00
Mike Auty
687da20cbc
Fix up a typo @npetroni pointed out.
2017-04-20 23:11:42 +01:00
Mike Auty
d6728fc2b4
Add in attributes and appropriate validation of TranslationLayers based on those attributes.
2017-02-10 13:57:44 +00:00
Mike Auty
1437e24b53
Add a note about _maxphyaddr.
2017-01-02 22:17:35 +00:00
Mike Auty
3aec66333e
Allow the vmware stacker to ask the FileLayer it's on top of what filename it had.
2016-12-29 15:10:21 +00:00
Mike Auty
bab71d4358
Refactor the progress_callback so it can be used in automagics.
...
Refactored the location of the progress_callback so that the UI can
specify it and ensure it is used by the automagics process as well as
the plugins themselves.
The progress_callback function has also had an optional description parameter
added, so that it can differentiated in case two scans happen
simultaneously.
2016-12-26 23:45:47 +00:00
Mike Auty
b54cb86c89
Allow automagics to use natives without a set of global natives.
...
This allows NativeSymbolTables to be appended to symbol_spaces and
uses this to let automagic modules (which have no existing symbol table
to rely on) to load natives for reading format data.
2016-12-26 22:49:51 +00:00
Mike Auty
943c625d5a
Ensure VMware fails back to flat with zero region metadata files.
2016-12-26 22:48:29 +00:00
Mike Auty
a4f27e73f4
Refactor the scanners to allow multithreaded searching.
2016-12-25 17:01:05 +00:00
Mike Auty
1c85db0c31
Add in multi-processing on TranslationLayer scan method.
2016-12-25 13:45:08 +00:00
Mike Auty
60bf657895
Allow pickling of FileLayers by unsetting the file (it'll be reopened as soon as a read happens).
2016-12-25 00:02:42 +00:00
Mike Auty
aad6aa9b4d
Rewrite the intel scanning engine (and fix up the internal _translate function to include the mapped layer).
2016-12-21 16:53:52 +00:00
Mike Auty
cad0cf2d4f
Improve vmware segment calculations.
2016-12-14 01:31:22 +00:00
Mike Auty
eec68707bd
Improve the vmware layer (not sure it's finished yet though).
2016-12-14 00:28:05 +00:00
Mike Auty
fa427d2a2b
Add in the starts of a vmware layer and stacker.
2016-12-13 02:21:15 +00:00
Mike Auty
21c4e4bdef
Refactor the segmented layer out of lime for use with other layers.
2016-12-11 19:13:35 +00:00
Mike Auty
13b1126b90
Guard against no progress_callback in intel layer.
2016-12-08 16:03:22 +00:00
Mike Auty
183ff67699
Attach the kernel virtual offset to intel spaces.
2016-12-07 16:00:31 +00:00
Mike Auty
20d0e83e1d
Ensure we don't ever round down when log-ing.
2016-12-07 00:05:47 +00:00
Mike Auty
734b9a68b8
Ensure we use math.log correctly.
...
This is unlikely to ever happen (since we use python3)
but still just as well to implement in case something else
similar comes up.
https://www.willmcgugan.com/blog/tech/post/finding-the-first-bit-set-with-python/
2016-12-06 23:40:21 +00:00
Mike Auty
a0f7c81128
Reveal the register size in intel spaces.
2016-12-04 02:14:54 +00:00
Mike Auty
d2fcc1d4b0
Add in optimization technique to reduce the number of pages that are scanned more than once.
2016-11-19 23:34:08 +00:00
Mike Auty
c9abd290ec
Remove an unnecessary test from the intel scan loop.
2016-11-19 18:05:41 +00:00
Mike Auty
f707c0c4e9
Add in first shot at an 'efficient' virtual memory scanner. Uses page misses to calculate skipping distances.
2016-11-18 17:14:48 +00:00
Mike Auty
096e2f4610
Make the paged exceptions more useful at knowing how big an area is unavailable.
2016-11-18 11:40:48 +00:00
Mike Auty
667014a1fd
Attempt to standardize error message display.
2016-10-30 17:51:45 +00:00
Mike Auty
824ad5b143
Add in the LimeStacker, and fix the lack of appropriate information on LimeLayer creation.
2016-08-26 08:35:09 +01:00
Mike Auty
37dcbb88b0
Rework lime to use the new layer constructor, and factor out the header check.
2016-08-26 08:30:22 +01:00
Mike Auty
3dc161f9b4
Rework Translation Layers to tie more closely to configurations
...
This change is quite signficant, and requires that TranslationLayers
get all additional parameters that they need through their requirements.
These are now automatically enumerated and populated on object
construction based on the requirements, so should not require lots of
repetitive filling out of fields.
It does come with the downside that TranslationLayers can only be
contructed with a context (and appropiate config), but TLs in particular
always require a context (to contain the base layer) and blank configs
can be constructed relatively easily (convenience functions can be added
if necessary).
This allows configuration trees to be built up, and their configs
spliced into an existing config (as if it were being loaded from a
file).
Not all ConstructableRequirements use this method, since SymbolTables
(for example) do not have access to the context or config_path in order
to get to any parameters stored in the context's config. They therefore
are still passed their requirement values as __init__ parameters
instead.
2016-08-22 02:20:10 +01:00
Mike Auty
d3e63fbc1a
Rename update_configuration to build_configuration and return a standalone config.
2016-08-22 02:18:23 +01:00
Mike Auty
e54505701f
Refactor Memory into an interface.
2016-08-21 21:26:21 +01:00
Mike Auty
462295fc4b
Ensure lime is properly imported, so that its classes get picked up by the plugin system.
2016-08-16 10:55:27 +01:00
Mike Auty
ca61cf10b9
Refactor the context back into the ConfigurableInterface.
...
This also adds support for manually constructed configurables to
populate the config tree in the current context.
I'm still toying around with this though, I need to figure out
what to do with optional values and think the whole thing through
to make sure it's worthwhile.
2016-08-14 21:28:43 +01:00
Mike Auty
8089a03f81
Convert the private file attribute to a property to allow for cloning.
2016-08-14 01:09:36 +01:00
Mike Auty
e02feed16e
Python 3 has a sane super() implementation (no arguments), so convert to using that.
2016-08-14 00:55:02 +01:00
Mike Auty
43aba96e3f
Minor pylint (alphabetical imports) and refactoring the construct class to work for symbols.
2016-07-30 01:58:29 +01:00
Mike Auty
d3a226d7a8
Remove constraints as a thing, they'll need to come back, but in a different form.
2016-07-30 01:52:44 +01:00
Mike Auty