This website requires JavaScript.
Explore
Help
Sign In
Pathogen
/
monolith
Watch
2
Star
0
Fork
You've already forked monolith
0
Code
Issues
Pull Requests
Projects
Releases
Wiki
Activity
Multi-source OSINT data collection and parallel processing tool. Indexes 4chan, Discord and IRC, reorganizes the data into a common format, annotates language, sentiment and tokens in multiple threads, and outputs the results to Elasticsearch.
530
Commits
1
Branch
1
Tag
8
MiB
Python
99.1%
Dockerfile
0.4%
Makefile
0.3%
Shell
0.2%
0ab67becff
Go to file
HTTPS
Download ZIP
Download TAR.GZ
Download BUNDLE
Clone in VS Code
Cite this repository
APA
BibTeX
Cancel
Mark Veidemanis
0ab67becff
Give option for only crawling some boards
2022-12-22 07:20:26 +00:00
docker
Add pathogen network
2022-11-23 18:23:20 +00:00
legacy
Fix mapping and make Threshold talk to SSDB
2022-11-22 21:42:35 +00:00
processing
Fully implement Elasticsearch indexing
2022-11-22 20:15:02 +00:00
schemas
Implement threshold writing to Redis and manticore ingesting from Redis
2022-09-07 07:20:30 +01:00
sources
Give option for only crawling some boards
2022-12-22 07:20:26 +00:00
.gitignore
Update gitignore
2022-10-21 11:53:28 +01:00
.pre-commit-config.yaml
Add ripsecrets to pre-commit hook
2022-11-03 07:20:30 +00:00
Makefile
Clean up docker environment
2022-10-19 16:45:18 +01:00
db.py
Pre-create meta index
2022-11-23 19:02:31 +00:00
env.example
Update env example file
2022-11-22 20:17:40 +00:00
monolith.py
Run ingest task first
2022-12-22 10:11:48 +00:00
requirements.txt
Clean up legacy and debugging code
2022-11-22 07:20:27 +00:00
util.py
Implement sentiment/NLP annotation and optimise processing
2022-09-16 17:09:49 +01:00