Multi-source OSINT data collection and parallel processing tool. Indexes 4chan, Discord and IRC, reorganizes the data into a common format, annotates language, sentiment and tokens in multiple threads, and outputs the results to Elasticsearch.
Go to file
Mark Veidemanis 027c43b60a
Don't muddle up the topics when sending Kafka batches
2022-09-20 23:03:02 +01:00
docker Update DirectMemorySize to be 1.5GB 2022-09-19 21:51:07 +01:00
legacy Treat text fields as string and try beta Kibana image 2022-09-12 08:27:13 +01:00
processing Make CPU threads configurable 2022-09-20 22:29:13 +01:00
schemas Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
sources Don't muddle up the topics when sending Kafka batches 2022-09-20 23:03:02 +01:00
.gitignore Add config directories to gitignore 2022-09-08 09:45:18 +01:00
.pre-commit-config.yaml Reinstate Redis cache 2022-09-04 21:38:53 +01:00
db.py Don't muddle up the topics when sending Kafka batches 2022-09-20 23:03:02 +01:00
docker-compose.yml Make performance settings configurable 2022-09-20 22:22:13 +01:00
env.example Document new PROCESS_THREADS setting in example file 2022-09-20 22:43:04 +01:00
environment Implement Apache Druid/Kafka and Metabase 2022-09-13 22:17:32 +01:00
event_log.txt Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
monolith.py Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
requirements.txt Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
util.py Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00