XF/monolith
2
0
Fork 0
Code Issues 2 Pull Requests Projects Releases Wiki Activity
485 Commits 1 Branch 1 Tag 8 MiB
cc6340acab
Commit Graph

485 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mark Veidemanis
cc6340acab
Add persistent Redis data store and copy over Druid config to production 2022-10-04 20:26:58 +01:00
Mark Veidemanis
7b73229d5a
Add Apache Superset and fix Druid resource usage 2022-10-04 20:17:04 +01:00
Mark Veidemanis
35ba2cc947
Add postgres config to Metabase 2022-10-02 14:29:40 +01:00
Mark Veidemanis
817bfd8835
Time stuff and switch to gensim for tokenisation 2022-10-01 14:46:45 +01:00
Mark Veidemanis
40cf0c6430
Remove commented debug code 2022-09-30 07:22:22 +01:00
Mark Veidemanis
63081f68b7
Use only one Redis key for the queue to make chunk size more precise for thread allocation 2022-09-30 07:22:22 +01:00
Mark Veidemanis
5992498493
Remove ujson 2022-09-30 15:30:34 +01:00
Mark Veidemanis
328db4a6da
Reformat 2022-09-30 15:23:00 +01:00
Mark Veidemanis
c5c834da82
Add config file to Turnilo 2022-09-27 08:30:28 +01:00
Mark Veidemanis
a8dbabd85e
Implement uvloop 2022-09-23 07:20:30 +01:00
Mark Veidemanis
56b5c85fac
Print Ingest settings on start 2022-09-23 08:32:29 +01:00
Mark Veidemanis
fc7450c33a
Make debug output cleaner 2022-09-22 17:39:29 +01:00
Mark Veidemanis
0e9a016e2a
Fix indexer options 2022-09-22 17:39:18 +01:00
Mark Veidemanis
763501d1ee
Fix Java variable in indexer parameters 2022-09-22 08:41:59 +01:00
Mark Veidemanis
40a215e6ec
Decrease memory requirements further and switch Kafka image 2022-09-21 21:11:13 +01:00
Mark Veidemanis
7abf9a00cb
Set Kafka max heap size 2022-09-21 20:26:05 +01:00
Mark Veidemanis
bd3f1ecd53
Set max memory for Metabase 2022-09-21 14:39:11 +01:00
Mark Veidemanis
64ebcedd76
Remove debugging code and fix regex substitution 2022-09-21 12:48:54 +01:00
Mark Veidemanis
3d293daad3
Change dev container names 2022-09-21 12:09:18 +01:00
Mark Veidemanis
00890860c0
Change prod container names 2022-09-21 12:08:29 +01:00
Mark Veidemanis
b0efaeef90
Remove prod compose comment 2022-09-21 12:04:54 +01:00
Mark Veidemanis
d6d19625f3
Remove commented code for debugging 2022-09-21 10:02:05 +01:00
Mark Veidemanis
cf4aa45663
Normalise fields in processing and remove invalid characters 2022-09-21 10:01:12 +01:00
Mark Veidemanis
48e4c07959
Make production volumes point to external storage 2022-09-21 10:00:48 +01:00
Mark Veidemanis
027c43b60a
Don't muddle up the topics when sending Kafka batches 2022-09-20 23:03:02 +01:00
Mark Veidemanis
e0803d4934
Document new PROCESS_THREADS setting in example file 2022-09-20 22:43:04 +01:00
Mark Veidemanis
6de17063a2
Make CPU threads configurable 2022-09-20 22:29:13 +01:00
Mark Veidemanis
2c5133a546
Make performance settings configurable 2022-09-20 22:22:13 +01:00
Mark Veidemanis
24929a5fbb
Set memory size to 2.5GB 2022-09-08 07:20:30 +01:00
Mark Veidemanis
f336d96268
Update DirectMemorySize to be 1.5GB 2022-09-19 21:51:07 +01:00
Mark Veidemanis
315e477916
Make MaxDirectMemory 0.5*cores 2022-09-19 19:15:57 +01:00
Mark Veidemanis
006677819d
Make max memory size 512m 2022-09-19 19:10:33 +01:00
Mark Veidemanis
93a0be98ce
Further decrease Druid memory requirements 2022-09-19 17:07:15 +01:00
Mark Veidemanis
14322f5090
Bump production Kafka healthcheck timeout 2022-09-19 11:18:52 +01:00
Mark Veidemanis
d94da5ac5c
Decrease production Druid max memory size 2022-09-19 10:51:34 +01:00
Mark Veidemanis
a1382ee46d
Increase Kafka retries 2022-09-19 10:48:29 +01:00
Mark Veidemanis
5e6b962ea8
Change Metabase port 2022-09-18 13:15:10 +01:00
Mark Veidemanis
e8dd847b36
Add docker environment file 2022-09-18 13:05:08 +01:00
Mark Veidemanis
d68bcfaebd
Update production compose 2022-09-18 13:04:08 +01:00
Mark Veidemanis
ebfa06e8d6
Reformat comment 2022-09-18 13:02:06 +01:00
Mark Veidemanis
3ed382ec13
Implement restricted sources 2022-09-18 13:01:19 +01:00
Mark Veidemanis
dab5e81715
Fix merge conflict 2022-09-16 17:45:24 +01:00
Mark Veidemanis
143f2a0bf0
Implement sentiment/NLP annotation and optimise processing 2022-09-16 17:09:49 +01:00
Mark Veidemanis
4ea77ac543
Properly process Redis buffered messages and ingest into Kafka 2022-09-14 18:32:32 +01:00
Mark Veidemanis
fec0d379a6
Ingest into Kafka and queue messages better 2022-09-13 22:17:46 +01:00
Mark Veidemanis
3c2adfc16e
Implement Apache Druid/Kafka and Metabase 2022-09-13 22:17:32 +01:00
Mark Veidemanis
4c6fe87b88
Switch to latest image for dev docker-compose 2022-09-13 09:20:43 +01:00
Mark Veidemanis
79a430be04
Begin implementing Apache Druid 2022-09-08 07:20:30 +01:00
Mark Veidemanis
baea6aebeb
Use stable after all 2022-09-08 07:20:30 +01:00
Mark Veidemanis
eaecc5cdbe
Switch production image back to dev 2022-09-08 07:20:30 +01:00