174 lines
9.7 KiB
Markdown
174 lines
9.7 KiB
Markdown
# GIA — Agent Knowledge Base
|
|
|
|
## Overview
|
|
|
|
GIA is a multi-transport communication platform bridging Signal, WhatsApp, XMPP, and Instagram through a Django web interface. It provides message relay, AI-powered workspace analysis, compose UX, and OSINT search. Stack: Python 3.11, Django 4.x, HTMX, Bulma CSS, SQLite, Redis, Docker Compose. Async runtime uses asyncio + uvloop.
|
|
|
|
## Structure
|
|
|
|
```
|
|
GIA/
|
|
├── app/ # Django project config (settings, urls, asgi, wsgi)
|
|
│ ├── settings.py # Main settings (imports local_settings.py at bottom)
|
|
│ ├── local_settings.py # Env-driven overrides (secrets, feature flags)
|
|
│ ├── urls.py # All URL routing (single flat file, no includes)
|
|
│ └── asgi.py # ASGI entrypoint with WebSocket routing
|
|
├── core/ # ALL application logic lives here
|
|
│ ├── models.py # All models (~1600 lines) — User, Person, AI, Message, etc.
|
|
│ ├── forms.py # Django ModelForms using RestrictedFormMixin from mixins
|
|
│ ├── admin.py # Admin registrations
|
|
│ ├── clients/ # Transport service adapters
|
|
│ │ ├── __init__.py # ClientBase ABC (start, message_received, etc.)
|
|
│ │ ├── transport.py # Shared transport layer — attachment prep, send, runtime state
|
|
│ │ ├── whatsapp.py # WhatsApp client via Neonize (~3100 lines)
|
|
│ │ ├── signal.py # Signal client via signal-cli REST API
|
|
│ │ ├── xmpp.py # XMPP client via slixmpp
|
|
│ │ ├── instagram.py # Instagram client via aiograpi
|
|
│ │ └── gateway.py # Gateway HTTP helpers
|
|
│ ├── messaging/ # Message processing pipeline
|
|
│ │ ├── ai.py # OpenAI integration (AsyncOpenAI)
|
|
│ │ ├── history.py # Prompt window builder with adaptive limits
|
|
│ │ ├── media_bridge.py # Media attachment resolution
|
|
│ │ ├── analysis.py # Conversation analysis
|
|
│ │ ├── natural.py # Natural language processing
|
|
│ │ ├── replies.py # Reply generation
|
|
│ │ └── utils.py # Message formatting helpers
|
|
│ ├── modules/
|
|
│ │ └── router.py # UnifiedRouter — orchestrates all transport clients
|
|
│ ├── views/ # Django views (class-based)
|
|
│ │ ├── compose.py # Compose UX (~3400 lines) — send, drafts, thread, media
|
|
│ │ ├── workspace.py # AI workspace (~5200 lines) — insights, mitigation, patterns
|
|
│ │ ├── osint.py # OSINT/search interface
|
|
│ │ └── ... # CRUD views for people, groups, sessions, etc.
|
|
│ ├── lib/prompts/ # AI persona prompt templates
|
|
│ ├── realtime/ # WebSocket handlers (compose thread)
|
|
│ ├── templates/ # Django templates (75 files, partials/ heavy)
|
|
│ ├── management/commands/ # ur (unified router), scheduling
|
|
│ └── util/logs.py # Custom colored logger — use logs.get_logger("name")
|
|
├── Makefile # Docker Compose orchestration commands
|
|
├── docker-compose.yml # Services: app, asgi, ur, scheduling, redis, signal-cli
|
|
├── Dockerfile # Python 3.11, venv at /venv
|
|
├── requirements.txt # Pinned deps (django, openai, neonize, slixmpp, etc.)
|
|
├── stack.env # Runtime env vars (from stack.env.example)
|
|
└── LLM_CODING_STANDARDS.md # Project-specific coding rules (READ THIS)
|
|
```
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
# All commands run via Docker Compose with stack.env
|
|
make build # Build Docker images
|
|
make run # Start all services (quadlet manager)
|
|
make stop # Stop all services
|
|
make log # Tail logs
|
|
make compose-run # Start via docker-compose directly
|
|
make compose-stop # Stop via docker-compose
|
|
make compose-log # Tail via docker-compose
|
|
|
|
# Database
|
|
make migrate # Run Django migrations
|
|
make makemigrations # Generate new migrations
|
|
make auth # Create superuser
|
|
|
|
# Testing
|
|
make test # Run all tests
|
|
make test MODULES=core.tests # Run specific test module
|
|
# Inside container (or with venv activated):
|
|
python manage.py test core.tests -v 2 # All tests
|
|
python manage.py test core.tests.test_foo -v 2 # Single test module
|
|
python manage.py test core.tests.test_foo.TestBar -v 2 # Single class
|
|
python manage.py test core.tests.test_foo.TestBar.test_method -v 2 # Single test
|
|
|
|
# Service restarts after code changes
|
|
docker-compose restart ur # Restart unified router
|
|
docker-compose restart scheduling # Restart scheduler
|
|
# uWSGI auto-reloads for app/core code changes
|
|
```
|
|
|
|
## Code Style
|
|
|
|
### Formatting & Linting (pre-commit enforced)
|
|
- **Black**: Line length 88, excludes `core/migrations/`
|
|
- **isort**: Profile `black` (compatible grouping)
|
|
- **flake8**: Max line 88, ignores E203, E231, E501, E702, W291
|
|
- **djhtml**: Template indent 2 spaces (`-t 2`)
|
|
- **ripsecrets**: Scans for leaked credentials
|
|
|
|
### Imports
|
|
- Standard library first, then third-party, then Django, then project
|
|
- Project imports use absolute paths: `from core.models import Person`
|
|
- Within same package, relative OK: `from .models import User` (seen in admin/forms)
|
|
- Views import models explicitly: `from core.models import AI, Person, Message`
|
|
|
|
### Naming
|
|
- `snake_case` for functions, variables, modules
|
|
- `PascalCase` for classes (Django views, models)
|
|
- `UPPER_CASE` for module-level constants and Django settings
|
|
- Private helpers prefixed with `_`: `_safe_limit()`, `_service_key()`
|
|
- Service names always lowercase strings: `"signal"`, `"whatsapp"`, `"xmpp"`, `"instagram"`
|
|
|
|
### Logging
|
|
- Use `from core.util import logs` then `log = logs.get_logger("name")` for transport/messaging code
|
|
- Use `import logging; logger = logging.getLogger(__name__)` for views/models
|
|
- Info for lifecycle events, warning/error for failures, debug for high-volume traces
|
|
- Debug logs gated behind `GIA_DEBUG_LOGS` env var
|
|
|
|
### Views
|
|
- Class-based views inheriting `View` or `LoginRequiredMixin`
|
|
- CRUD views use `mixins.views.ObjectCreate/ObjectUpdate/ObjectDelete` from `django-crud-mixins`
|
|
- Forms use `RestrictedFormMixin` for user-scoped queryset filtering
|
|
- HTMX-driven partials in `core/templates/partials/`
|
|
|
|
### Models
|
|
- All models in single `core/models.py` (~1600 lines)
|
|
- UUIDs as primary keys for `Person` and related models
|
|
- `SERVICE_CHOICES` tuple for transport type fields
|
|
- Custom `User` model extending `AbstractUser` with billing fields
|
|
- Multi-tenant: most models have `user = ForeignKey(User)`
|
|
|
|
### Async Patterns
|
|
- Transport clients are async (`async def send_message_raw(...)`)
|
|
- Views bridge sync Django to async transport via `async_to_sync()`
|
|
- `orjson` for fast JSON serialization in transport layer
|
|
- Redis cache for runtime state, bridge maps, command queues
|
|
|
|
### Error Handling
|
|
- Standard `try/except` with specific exception types
|
|
- Django `ValidationError` for model validation
|
|
- `get_object_or_404()` in views for missing resources
|
|
- `HttpResponseBadRequest` / `HttpResponseNotFound` for view error responses
|
|
- No custom exception hierarchy — use built-in Django/Python exceptions
|
|
|
|
## LLM Coding Standards (from LLM_CODING_STANDARDS.md)
|
|
|
|
**MUST READ**: `LLM_CODING_STANDARDS.md` contains binding project rules. Key points:
|
|
|
|
- Fix root causes; don't paper over with UI-only patches
|
|
- Keep behavior symmetric across all transports where protocol permits
|
|
- Centralize shared logic — no copy/paste service forks
|
|
- Shared attachment prep goes through `core/clients/transport.py`
|
|
- Never inject internal blob links as relay body text for attachment-only messages
|
|
- After changing `core/clients/*` or router/relay/transport: restart runtime (`make stop && make run`)
|
|
- Logging: lifecycle at info, failures at warning/error, high-volume at debug
|
|
- Debug diagnostics must be gated (e.g. `WHATSAPP_DEBUG`) and removable in one patch
|
|
- When touching large files (2000+ lines): extract minimal reusable helpers, add docstrings
|
|
- Update `INSTALL.md` and `README.md` when operational commands/env requirements change
|
|
|
|
## Anti-Patterns
|
|
|
|
- **DO NOT** create separate transport-specific media pipelines — use `transport.prepare_outbound_attachments()`
|
|
- **DO NOT** add `TODO`/`FIXME` comments — codebase is currently clean of them
|
|
- **DO NOT** use `print()` — use the logging system via `logs.get_logger()`
|
|
- **DO NOT** modify `core/migrations/` files — Black/linting excludes them for a reason
|
|
- **DO NOT** commit `stack.env`, `db.sqlite3`, or any secrets — `ripsecrets` pre-commit hook will block
|
|
- **DO NOT** add new models outside `core/models.py` — all models live in one file
|
|
- **DO NOT** use type suppression or ignore runtime errors silently
|
|
|
|
## Key Architecture Notes
|
|
|
|
- **Unified Router** (`core/modules/router.py`): Management command `python manage.py ur` runs the event loop with all transport clients. Each client inherits `ClientBase` ABC.
|
|
- **Transport Layer** (`core/clients/transport.py`): Shared cache-backed runtime state, command queuing, and attachment prep. All outbound media goes through `prepare_outbound_attachments()`.
|
|
- **Settings Chain**: `app/settings.py` → imports `app/local_settings.py` (wildcard `*`) → env vars from `stack.env`. Feature flags: `WHATSAPP_ENABLED`, `INSTAGRAM_ENABLED`, `COMPOSE_WS_ENABLED`.
|
|
- **Services in docker-compose**: `app` (uWSGI), `asgi` (uvicorn for WebSockets), `ur` (unified router), `scheduling` (APScheduler), `redis`, `signal-cli-rest-api`.
|
|
- **No test suite currently**: `core/tests.py` is empty scaffold; `core/tests/` has only `__init__.py`. Tests run via `make test MODULES=...` but need to be written.
|