Commit Graph

27 Commits

Author SHA1 Message Date
Mark Veidemanis aea1c7faf6 Use one image for all the Druid services 2022-10-04 21:30:17 +01:00
Mark Veidemanis d7d9958e54 Add persistent Redis data store and copy over Druid config to production 2022-10-04 20:26:58 +01:00
Mark Veidemanis 464c831686 Add Apache Superset and fix Druid resource usage 2022-10-04 20:17:04 +01:00
Mark Veidemanis 5ad6cd0354 Add postgres config to Metabase 2022-10-02 14:29:40 +01:00
Mark Veidemanis 02ff44a6f5 Use only one Redis key for the queue to make chunk size more precise for thread allocation 2022-09-30 07:22:22 +01:00
Mark Veidemanis 2dd2360b4f Add config file to Turnilo 2022-09-27 08:30:28 +01:00
Mark Veidemanis 395dfb1e7b Decrease memory requirements further and switch Kafka image 2022-09-21 21:11:13 +01:00
Mark Veidemanis ee79762c73 Set Kafka max heap size 2022-09-21 20:26:05 +01:00
Mark Veidemanis e58b9960b2 Set max memory for Metabase 2022-09-21 14:39:11 +01:00
Mark Veidemanis 4a60dec964 Remove debugging code and fix regex substitution 2022-09-21 12:48:54 +01:00
Mark Veidemanis 9ee55a720b Change dev container names 2022-09-21 12:09:18 +01:00
Mark Veidemanis 40a0c2d22e Make performance settings configurable 2022-09-20 22:22:13 +01:00
Mark Veidemanis cb11ce9b12 Fix merge conflict 2022-09-16 17:45:24 +01:00
Mark Veidemanis a89b5a8b6f Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
Mark Veidemanis f432e9b29e Properly process Redis buffered messages and ingest into Kafka 2022-09-14 18:32:32 +01:00
Mark Veidemanis 47c5f89914 Implement Apache Druid/Kafka and Metabase 2022-09-13 22:17:32 +01:00
Mark Veidemanis 68fd5fa230 Switch to latest image for dev docker-compose 2022-09-13 09:20:43 +01:00
Mark Veidemanis fd90c233c2 Begin implementing Apache Druid 2022-09-08 07:20:30 +01:00
Mark Veidemanis 9cf4e945d1 Set dev image back to the default 2022-09-12 08:43:18 +01:00
Mark Veidemanis 04b5dec843 Treat text fields as string and try beta Kibana image 2022-09-12 08:27:13 +01:00
Mark Veidemanis c2bdb3fd15 Reformat 2022-09-07 07:20:30 +01:00
Mark Veidemanis 79b1bee9e4 Implement ingesting to Redis from Threshold 2022-09-07 07:20:30 +01:00
Mark Veidemanis cd5eb61455 Improve DB performance with caching 2022-09-05 07:20:30 +01:00
Mark Veidemanis 60c43b4eb5 Run processing in thread 2022-09-04 21:29:00 +01:00
Mark Veidemanis db23b31f30 Implement aiohttp 2022-09-04 19:44:25 +01:00
Mark Veidemanis f7860bf08b Begin implementing aiohttp 2022-09-04 13:47:32 +01:00
Mark Veidemanis 734a2b7879 Implement running Discord and 4chan gathering simultaneously 2022-09-02 22:30:45 +01:00