Discord/4chan ingest tool buffered by Redis sending streams to Kafka. Container definitions for Apache Druid, Metabase and Turnilo.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Mark Veidemanis 508b00e471
Pre-create meta index
2 weeks ago
docker Add pathogen network 2 weeks ago
legacy Fix mapping and make Threshold talk to SSDB 2 weeks ago
processing Fully implement Elasticsearch indexing 2 weeks ago
schemas Implement threshold writing to Redis and manticore ingesting from Redis 3 months ago
sources Improve memory usage and fix 4chan crawler 2 months ago
.gitignore Update gitignore 2 months ago
.pre-commit-config.yaml Add ripsecrets to pre-commit hook 1 month ago
Makefile Clean up docker environment 2 months ago
db.py Pre-create meta index 2 weeks ago
env.example Update env example file 2 weeks ago
monolith.py Use only one Redis key for the queue to make chunk size more precise for thread allocation 2 months ago
requirements.txt Clean up legacy and debugging code 2 weeks ago
util.py Implement sentiment/NLP annotation and optimise processing 3 months ago