Commit Graph
16 Commits
Author SHA1 Message Date
Mike Auty 1f4498be37 MultiRegExScanner: Results are always bytes and never strings 2019-11-27 11:30:14 +00:00
Mike Auty b0b868d79c Typing: General typing fixes across the tree 2019-11-27 11:30:14 +00:00
Mike Auty e0b9d72c84 Update the license URL and LICENSE.txt 2019-09-25 21:44:05 +01:00
Mike Auty e922cef316 Improve docstrings for all plugins, and reformat all docstrings. 2019-09-07 22:59:54 +01:00
Mike Auty 9cde94bceb Change the license blurb on all files. 2019-08-27 23:11:00 +01:00
Mike Auty 9fadee1a17 Add locking for threaded operation. 2019-03-02 18:07:48 +00:00
Mike Auty c64ed6a8dd Remove validity checking in favour of static-type checking.
This removes the assertions that checked plugin devs didn't do anything
overly bad in favour of using mypy to ensure appropriate type-checking.

It also moves the ProgressCallback typing information to constants in
order to avoid circular imports (since constants doesn't import
anything).
2019-01-20 02:35:49 +00:00
Mike Auty fdc6efc4ad Add in the LICENSE and the README. 2018-12-19 23:55:03 +00:00
Mike Auty 29d41470a4 Mass reformat of typing imports
Relented on the strict import of direct objects/classes for the typing
module only.  Typing module components can be directly imported because
it makes the code really painful to read and write otherwise.

This is still in-line with the python style guide adopted from Google at
http://google.github.io/styleguide/pyguide.html section 2.2.
2018-12-16 13:04:22 +00:00
Mike Auty a07691b04f Ensure scanners don't return duplicates in the overlap. 2018-05-13 20:55:54 +01:00
Mike Auty ccd00afdb2 Add in new multistring scanner based on re
Unfortunately in tests it turned out re was faster than a python
implementation of Wu-Manber.  The Wu-Manber code has been left (because
why not) but it's not really advantageous over the re builtin since
it's compiled in C.
2018-01-24 20:22:00 +00:00
Mike Auty 1dcad91126 Add in type annotations for scanners. 2017-12-13 20:48:52 +00:00
Mike Auty f9375d0797 Reduce pointless memory usage, and remove old imports and debugging exception handlers. 2017-07-16 01:23:31 +01:00
Mike Auty 0ca78bc890 Add an efficient multi-string search module (based on Wu-Manber 1994).
Empirically this seems far quicker than either Aho-Corasick or
Commentz-Walter for our dataset (very many long search terms).
2017-07-16 01:13:49 +01:00
Mike Auty 2b718f104d Remove imports that will break the code. 2017-07-14 01:15:52 +01:00
Mike Auty dd7c9d777a Refactor the scanners to allow for better organization. 2017-07-13 01:30:01 +01:00