Multi-source OSINT data collection and parallel processing tool. Indexes 4chan, Discord and IRC, reorganizes the data into a common format, annotates language, sentiment and tokens in multiple threads, and outputs the results to Elasticsearch.
Go to file
Mark Veidemanis fed3cdbf5a Remove indexer block to attempt to prevent Manticore DB crash 2022-09-08 07:20:30 +01:00
docker Remove indexer block to attempt to prevent Manticore DB crash 2022-09-08 07:20:30 +01:00
legacy Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
schemas Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
sources Reformat 2022-09-07 07:20:30 +01:00
.gitignore Add config directories to gitignore 2022-09-08 09:45:18 +01:00
.pre-commit-config.yaml Reinstate Redis cache 2022-09-04 21:38:53 +01:00
db.py Add 4chan update message type to main types 2022-09-07 07:20:30 +01:00
docker-compose.yml Reformat 2022-09-07 07:20:30 +01:00
monolith.py Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
requirements.txt Add aioredis 2022-09-08 09:44:27 +01:00
util.py Run processing in thread 2022-09-04 21:29:00 +01:00