Multi-source OSINT data collection and parallel processing tool. Indexes 4chan, Discord and IRC, reorganizes the data into a common format, annotates language, sentiment and tokens in multiple threads, and outputs the results to Elasticsearch.
Go to file
Mark Veidemanis 0ab67becff
Give option for only crawling some boards
2022-12-22 07:20:26 +00:00
docker Add pathogen network 2022-11-23 18:23:20 +00:00
legacy Fix mapping and make Threshold talk to SSDB 2022-11-22 21:42:35 +00:00
processing Fully implement Elasticsearch indexing 2022-11-22 20:15:02 +00:00
schemas Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
sources Give option for only crawling some boards 2022-12-22 07:20:26 +00:00
.gitignore Update gitignore 2022-10-21 11:53:28 +01:00
.pre-commit-config.yaml Add ripsecrets to pre-commit hook 2022-11-03 07:20:30 +00:00
Makefile Clean up docker environment 2022-10-19 16:45:18 +01:00
db.py Pre-create meta index 2022-11-23 19:02:31 +00:00
env.example Update env example file 2022-11-22 20:17:40 +00:00
monolith.py Run ingest task first 2022-12-22 10:11:48 +00:00
requirements.txt Clean up legacy and debugging code 2022-11-22 07:20:27 +00:00
util.py Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00