Pathogen
/
monolith
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
441 Commits 1 Branch 1 Tag 8 MiB
Commit Graph

441 Commits

Author SHA1 Message Date
Mark Veidemanis 4ea77ac543
Properly process Redis buffered messages and ingest into Kafka 2022-09-14 18:32:32 +01:00
Mark Veidemanis fec0d379a6
Ingest into Kafka and queue messages better 2022-09-13 22:17:46 +01:00
Mark Veidemanis 3c2adfc16e
Implement Apache Druid/Kafka and Metabase 2022-09-13 22:17:32 +01:00
Mark Veidemanis 79a430be04
Begin implementing Apache Druid 2022-09-08 07:20:30 +01:00
Mark Veidemanis baea6aebeb
Use stable after all 2022-09-08 07:20:30 +01:00
Mark Veidemanis eaecc5cdbe
Switch production image back to dev 2022-09-08 07:20:30 +01:00
Mark Veidemanis 764e36ef14
Lower memory requirements to prevent crashes 2022-09-08 07:20:30 +01:00
Mark Veidemanis 50a873dbba
Set dev image back to the default 2022-09-12 08:43:18 +01:00
Mark Veidemanis 21182629b4
Treat text fields as string and try beta Kibana image 2022-09-12 08:27:13 +01:00
Mark Veidemanis dfd71b6c64
Add Mysql port to ports instead of expose 2022-09-10 13:20:06 +01:00
Mark Veidemanis 1b0817b047
Expose the Mysql port 2022-09-10 13:16:19 +01:00
Mark Veidemanis 0ba4929294
Use dev image of manticore 2022-09-10 12:03:45 +01:00
Mark Veidemanis caded433b7
Remove indexer block to attempt to prevent Manticore DB crash 2022-09-08 07:20:30 +01:00
Mark Veidemanis bf802d7fdf
Reformat 2022-09-07 07:20:30 +01:00
Mark Veidemanis 89328a827a
Raise open files limit for Redis 2022-09-07 07:20:30 +01:00
Mark Veidemanis 32249a1d99
Add 4chan update message type to main types 2022-09-07 07:20:30 +01:00
Mark Veidemanis cdd12cd082
Implement threshold writing to Redis and manticore ingesting from Redis 2022-09-07 07:20:30 +01:00
Mark Veidemanis 137299fe9e
Add config directories to gitignore 2022-09-08 09:45:18 +01:00
Mark Veidemanis 2aedcf77a0
Add aioredis 2022-09-08 09:44:27 +01:00
Mark Veidemanis 49784dfbe5
Implement ingesting to Redis from Threshold 2022-09-07 07:20:30 +01:00
Mark Veidemanis a6b5348224
Config relative to Git dir 2022-09-05 07:20:30 +01:00
Mark Veidemanis d0fe2baafe
Store persistent database elsewhere 2022-09-05 07:20:30 +01:00
Mark Veidemanis e092327932
Improve DB performance with caching 2022-09-05 07:20:30 +01:00
Mark Veidemanis 8b9ad05089
Reformat legacy project 2022-09-05 07:20:30 +01:00
Mark Veidemanis 6b082adeb2
Merge branch 'threshold' 2022-09-06 12:50:25 +01:00
Mark Veidemanis bd9f9378cf
Moved files to subdirectory 2022-09-06 12:50:09 +01:00
Mark Veidemanis 62fe03a6cb
Increase thread delay time 2022-09-05 07:20:30 +01:00
Mark Veidemanis 297bbbe035
Alter schemas and 4chan performance settings 2022-09-05 07:20:30 +01:00
Mark Veidemanis ed7c439b56
Remove some debugging code 2022-09-05 07:20:30 +01:00
Mark Veidemanis ecb8079b5b
Change Python to 3.10 2022-09-05 07:20:30 +01:00
Mark Veidemanis 6811ce4af5
Update production env file path 2022-09-05 07:20:30 +01:00
Mark Veidemanis e34d281774
Remove development dotenv loading 2022-09-05 07:20:30 +01:00
Mark Veidemanis 91e18c60e6
Add debug statement 2022-09-05 07:20:30 +01:00
Mark Veidemanis 9c9d49dcd2
Reformat and set the net and channel for 4chan 2022-09-05 07:20:30 +01:00
Mark Veidemanis dcd648e1d2
Make crawler more efficient and implement configurable parameters 2022-09-05 07:20:30 +01:00
Mark Veidemanis 318a8ddbd5
Split thread list into chunks to save memory 2022-09-05 07:20:30 +01:00
Mark Veidemanis 20e22ae7ca
Reformat code 2022-09-04 21:40:04 +01:00
Mark Veidemanis 8feccbbf00
Reinstate Redis cache 2022-09-04 21:38:53 +01:00
Mark Veidemanis db46fea550
Run processing in thread 2022-09-04 21:29:00 +01:00
Mark Veidemanis 22cef33342
Implement aiohttp 2022-09-04 19:44:25 +01:00
Mark Veidemanis 663a26778d
Begin implementing aiohttp 2022-09-04 13:47:32 +01:00
Mark Veidemanis 36de004ee5
Implement running Discord and 4chan gathering simultaneously 2022-09-02 22:30:45 +01:00
Mark Veidemanis 2c3d83fe9a
Fix error when no email can be found 2022-08-27 11:19:28 +01:00
Mark Veidemanis d7adffb47f
Fix getting first relay when they are not sequential 2022-08-26 22:17:12 +01:00
Mark Veidemanis 4f4820818a
Log authentication messages 2022-08-16 23:01:42 +01:00
Mark Veidemanis 5cc38da00e
Implement deduplicating channels 2022-08-16 22:01:35 +01:00
Mark Veidemanis a4dae2a583
Switch to siphash 2022-08-18 07:20:30 +01:00
Mark Veidemanis 5f1667869f
Re-add fake messages 2022-08-15 19:49:21 +01:00
Mark Veidemanis 09a5cd14ad
Detect queries if nick and channel are the same 2022-08-15 19:24:42 +01:00
Mark Veidemanis 96de70aaf2
Add sinst fetch and fix message send logic 2022-08-15 19:15:12 +01:00