Commit Graph
26 Commits
Author SHA1 Message Date
Mike Auty 1f4498be37 MultiRegExScanner: Results are always bytes and never strings 2019-11-27 11:30:14 +00:00
Mike Auty b0b868d79c Typing: General typing fixes across the tree 2019-11-27 11:30:14 +00:00
Mike Auty e0b9d72c84 Update the license URL and LICENSE.txt 2019-09-25 21:44:05 +01:00
Mike Auty a248bdc804 Fix up old code and docstring indent issue. 2019-09-17 18:15:05 +01:00
Mike Auty ffc7682d61 Ensure that layers are automatically imported when needed. 2019-09-08 14:13:28 +01:00
Mike Auty e922cef316 Improve docstrings for all plugins, and reformat all docstrings. 2019-09-07 22:59:54 +01:00
Mike Auty 9cde94bceb Change the license blurb on all files. 2019-08-27 23:11:00 +01:00
Mike Auty 9fadee1a17 Add locking for threaded operation. 2019-03-02 18:07:48 +00:00
Mike Auty c64ed6a8dd Remove validity checking in favour of static-type checking.
This removes the assertions that checked plugin devs didn't do anything
overly bad in favour of using mypy to ensure appropriate type-checking.

It also moves the ProgressCallback typing information to constants in
order to avoid circular imports (since constants doesn't import
anything).
2019-01-20 02:35:49 +00:00
Mike Auty 0ee335237a Improve the protection of empty regular expressions (improves issue #61). 2019-01-18 11:27:24 +00:00
Mike Auty afdd8b657e Fix memory problems with banner_caches.
Since we stack using Mac layers as well as Linux layers, both sets of
banner caches are required.  As the cache builders are separate at the
moment, we previously ran banner searches without any banners.  This was
bad.  It inflated memory hugely and killed the buildbot.

We now bomb out on stacking a Linux or Mac layer unless we have banners
against which to match.
2018-12-30 15:13:10 +00:00
Mike Auty fdc6efc4ad Add in the LICENSE and the README. 2018-12-19 23:55:03 +00:00
Mike Auty 19572b6e1e Reformat all python files using yapf with custom volatility style. 2018-12-16 16:50:17 +00:00
Mike Auty 29d41470a4 Mass reformat of typing imports
Relented on the strict import of direct objects/classes for the typing
module only.  Typing module components can be directly imported because
it makes the code really painful to read and write otherwise.

This is still in-line with the python style guide adopted from Google at
http://google.github.io/styleguide/pyguide.html section 2.2.
2018-12-16 13:04:22 +00:00
Mike Auty 7a52ac9deb Fix a large number of typing issues.
There are several instances where mypy didn't detect

if blah is not None:
    blah = thing

and so were rewritten as:

blah = blah or thing
2018-06-04 01:25:02 +01:00
Mike Auty a07691b04f Ensure scanners don't return duplicates in the overlap. 2018-05-13 20:55:54 +01:00
Mike Auty 663cf3eff8 Just tidy up the multistring searchers slightly. 2018-01-25 11:27:59 +00:00
Mike Auty ccd00afdb2 Add in new multistring scanner based on re
Unfortunately in tests it turned out re was faster than a python
implementation of Wu-Manber.  The Wu-Manber code has been left (because
why not) but it's not really advantageous over the re builtin since
it's compiled in C.
2018-01-24 20:22:00 +00:00
Mike Auty 4bb1161490 Make WuManber a callable program. 2017-12-24 12:53:50 +00:00
Mike Auty 1dcad91126 Add in type annotations for scanners. 2017-12-13 20:48:52 +00:00
Mike Auty 807b5dcaf0 Ensure we throw an appropriate error if there's no patterns available. 2017-07-16 22:52:56 +01:00
Mike Auty c4fbaf248e Max the number of hash buckets needed precise, and explicitly linked to the hash function. 2017-07-16 01:27:20 +01:00
Mike Auty f9375d0797 Reduce pointless memory usage, and remove old imports and debugging exception handlers. 2017-07-16 01:23:31 +01:00
Mike Auty 0ca78bc890 Add an efficient multi-string search module (based on Wu-Manber 1994).
Empirically this seems far quicker than either Aho-Corasick or
Commentz-Walter for our dataset (very many long search terms).
2017-07-16 01:13:49 +01:00
Mike Auty 2b718f104d Remove imports that will break the code. 2017-07-14 01:15:52 +01:00
Mike Auty dd7c9d777a Refactor the scanners to allow for better organization. 2017-07-13 01:30:01 +01:00