Multi-source OSINT data collection and parallel processing tool. Indexes 4chan, Discord and IRC, reorganizes the data into a common format, annotates language, sentiment and tokens in multiple threads, and outputs the results to Elasticsearch.
Go to file
Mark Veidemanis cf4aa45663
Normalise fields in processing and remove invalid characters
2022-09-21 10:01:12 +01:00
docker Make production volumes point to external storage 2022-09-21 10:00:48 +01:00
legacy Treat text fields as string and try beta Kibana image 2022-09-12 08:27:13 +01:00
processing Normalise fields in processing and remove invalid characters 2022-09-21 10:01:12 +01:00
schemas Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
sources Normalise fields in processing and remove invalid characters 2022-09-21 10:01:12 +01:00
.gitignore Add config directories to gitignore 2022-09-08 09:45:18 +01:00
.pre-commit-config.yaml Reinstate Redis cache 2022-09-04 21:38:53 +01:00
db.py Normalise fields in processing and remove invalid characters 2022-09-21 10:01:12 +01:00
docker-compose.yml Make performance settings configurable 2022-09-20 22:22:13 +01:00
env.example Document new PROCESS_THREADS setting in example file 2022-09-20 22:43:04 +01:00
environment Implement Apache Druid/Kafka and Metabase 2022-09-13 22:17:32 +01:00
event_log.txt Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
monolith.py Normalise fields in processing and remove invalid characters 2022-09-21 10:01:12 +01:00
requirements.txt Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
util.py Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00