Multi-source OSINT data collection and parallel processing tool. Indexes 4chan, Discord and IRC, reorganizes the data into a common format, annotates language, sentiment and tokens in multiple threads, and outputs the results to Elasticsearch.
Go to file
Mark Veidemanis 395dfb1e7b Decrease memory requirements further and switch Kafka image 2022-09-21 21:11:13 +01:00
docker Decrease memory requirements further and switch Kafka image 2022-09-21 21:11:13 +01:00
legacy Remove debugging code and fix regex substitution 2022-09-21 12:48:54 +01:00
processing Remove debugging code and fix regex substitution 2022-09-21 12:48:54 +01:00
schemas Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
sources Remove commented code for debugging 2022-09-21 10:02:05 +01:00
.gitignore Add config directories to gitignore 2022-09-08 09:45:18 +01:00
.pre-commit-config.yaml Reinstate Redis cache 2022-09-04 21:38:53 +01:00
db.py Remove commented code for debugging 2022-09-21 10:02:05 +01:00
docker-compose.yml Decrease memory requirements further and switch Kafka image 2022-09-21 21:11:13 +01:00
env.example Document new PROCESS_THREADS setting in example file 2022-09-20 22:43:04 +01:00
environment Decrease memory requirements further and switch Kafka image 2022-09-21 21:11:13 +01:00
event_log.txt Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
monolith.py Remove commented code for debugging 2022-09-21 10:02:05 +01:00
requirements.txt Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
util.py Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00