Commit Graph

540 Commits

Author SHA1 Message Date
352909bec0
Begin implementing RTS process 2025-01-23 11:30:01 +00:00
1cc2ef629e
Automatic optimisation for ingest chunking based on processed messages 2025-01-23 11:29:48 +00:00
1aeadaf3b7
Stringify tokens and return message number from processing 2025-01-23 11:29:07 +00:00
ba8c33d8fc
Update socket and database URLs 2025-01-23 11:27:51 +00:00
2ef2249be7
Update container files to work with Podman 2025-01-23 11:26:58 +00:00
054e9caca0
Update to run with Podman 2024-12-29 17:35:57 +00:00
5ea4e5f460
Bump versions in pre-commit config 2023-02-09 07:20:13 +00:00
210237b50a
Update pre-commit versions 2023-02-09 07:20:13 +00:00
87b81ac236
Retry Redis ingest if it failed 2023-01-13 07:20:27 +00:00
02071758b5
Send messages to Neptune Redis via PubSub 2023-01-12 07:20:43 +00:00
0ab67becff
Give option for only crawling some boards 2022-12-22 07:20:26 +00:00
ebaf8c765d
Allow disabling modules in environment variables 2022-12-22 07:20:26 +00:00
ce2d7684bc
Run ingest task first 2022-12-22 10:11:48 +00:00
508b00e471
Pre-create meta index 2022-11-23 19:02:31 +00:00
9d1e4b44e8
Add pathogen network 2022-11-23 18:23:20 +00:00
371bce1094
Remove print statements 2022-11-22 21:43:56 +00:00
be0cf231b4
Fix mapping and make Threshold talk to SSDB 2022-11-22 21:42:35 +00:00
1993c8f1d2
Use Portainer Git directory for Redis config 2022-11-22 21:19:13 +00:00
9d35930b3b
Re-add Portainer Git dir 2022-11-22 21:15:19 +00:00
34346006ab
Remove leftover Docker files in legacy 2022-11-22 21:11:46 +00:00
42a3f5da03
Remove tarred Docker definition 2022-11-22 07:20:52 +00:00
42657aeee0
Lower Python version due to gensim incompatibility 2022-11-22 21:07:34 +00:00
1c34aa4a01
Clean up legacy and debugging code 2022-11-22 07:20:27 +00:00
6f3db61532
Add debugging statements for Portainer 2022-11-22 20:37:58 +00:00
3c18858c48
Use relative path for code 2022-11-22 20:35:13 +00:00
78c6ef96d2
Use relative path for build directory 2022-11-22 20:30:41 +00:00
d2a174c1c4
Remove Portainer Git volume 2022-11-22 20:28:44 +00:00
c53438d07b
Remove port variable 2022-11-22 20:17:51 +00:00
93353f34e7
Update env example file 2022-11-22 20:17:40 +00:00
6b1604b724
Remove old compose file 2022-11-22 20:17:28 +00:00
49f46c33ba
Fully implement Elasticsearch indexing 2022-11-22 20:15:02 +00:00
052631c71f
Remove infrastructure Docker definition 2022-11-22 18:41:29 +00:00
7edc231ea9
Remove Manticore and Superset stuff 2022-11-22 18:37:49 +00:00
44d6d90325
Update Druid spec 2022-11-21 18:59:53 +00:00
1c2ff41b56
Add ripsecrets to pre-commit hook 2022-11-03 07:20:30 +00:00
51a9b2af79
Improve memory usage and fix 4chan crawler 2022-10-21 07:20:30 +01:00
2d7b6268dd
Don't shadow previous iterator variable 2022-10-21 07:20:30 +01:00
e5b5268f5c
Add example Druid spec 2022-10-21 07:20:30 +01:00
dc1ed1fe10
Print the length of the flattened list in debug message 2022-10-21 07:20:30 +01:00
eaf9a3c937
Remove unused ssdb_data volume 2022-10-21 07:20:30 +01:00
054a7a3ccf
Don't mount the template directory 2022-10-21 07:20:30 +01:00
f774f4c2d2
Add some environment variables to control debug output 2022-10-21 07:20:30 +01:00
e32b330ef4
Switch to SSDB for message queueing 2022-10-21 11:53:29 +01:00
8c596ec516
Update gitignore 2022-10-21 11:53:28 +01:00
ab5e85c5c6 Begin switching away from Redis 2022-10-21 11:14:51 +01:00
7482064aee Clean up docker environment 2022-10-19 16:45:18 +01:00
dccbc6b158 Remove dependencies on infra stuff 2022-10-11 11:16:24 +01:00
8cc1a48a25 Separate out infra in production 2022-10-11 11:04:03 +01:00
83e8fb0e38 Remove event log file 2022-10-05 12:52:30 +01:00
64cf7d0d4a Set Superset directory relative to Portainer Git root 2022-10-04 21:43:16 +01:00