Security event processing pipeline built on Kafka, Redis, PostgreSQL, and FastAPI.
Ingests security events from Kafka, classifies them using a 3-tier analysis approach, and stores results in PostgreSQL. A React frontend provides a dashboard.
| Tier | Method | Purpose |
|---|---|---|
| 1 | Rule-based | Fast deterministic matching for known patterns |
| 2 | PyTorch classifier | Neural network for risk scoring and classification |
| 3 | DSPy-configured LLM | Detailed analysis of complex or ambiguous events |
The system picks the appropriate tier based on event characteristics. Nightly fine-tuning updates both the PyTorch model and DSPy configuration.
| Component | Technology |
|---|---|
| API | FastAPI |
| Queue | Kafka |
| Cache | Redis |
| Database | PostgreSQL + Alembic migrations |
| ML | PyTorch, DSPy, LiteLLM |
| Monitoring | Prometheus + Grafana |
| Auth | JWT (python-jose) |
| Frontend | React |
| Deployment | Docker Compose |
src/
api/ # FastAPI routes and middleware
models/ # SQLAlchemy + Pydantic models
services/ # Kafka, Redis, Prometheus, analytics
tasks/ # Nightly fine-tuning jobs
utils/ # Caching, error handling, IP utils
frontend/ # React dashboard
docker/ # Dockerfiles and Compose config
alembic/ # Database migrations
tests/ # 17 test modules
- Docker 20.10+
- Docker Compose 1.29+
git clone https://github.com/jmanhype/SecuStreamAI.git
cd SecuStreamAI
cp .env.example .env
# Edit .env with your OPENAI_API_KEY and other secrets
docker-compose -f docker/docker-compose.yaml up --buildAll configuration is via environment variables in .env. Key settings:
OPENAI_API_KEY-- required for tier-3 LLM analysisKAFKA_SERVER,REDIS_HOST,POSTGRES_HOST-- infrastructure endpointsSECRET_KEY-- JWT signing key
Prototype. The CI/CD workflows exist but reference placeholder Docker Hub paths (yourusername/secustreamai). The React frontend is a default Create React App scaffold with minimal customization. Test coverage exists across 17 modules but actual pass rates are unknown.
See LICENSE file.