🏗️ MyMindVentures AI Stack Architecture

Complete infrastructure visualization and design documentation

🔓 100% Open Source (MIT License)
✅ Fully Open Source: This diagram service is built with mermaid.js (MIT), Node.js (MIT), and runs completely self-hosted with no external paid services.

📊 System Overview

The complete MyMindVentures AI Stack architecture spanning 11 docker-compose stacks with 27 services

11
Compose Stacks
27
Services
6
Networks
15+
Volumes
graph TB Internet["🌍 Internet Users"] DNS["🔗 DNS"] Internet --> Caddy["🔐 EDGE: Caddy
Reverse Proxy + HTTPS
Rate Limiting
Security Headers"] DNS --> Caddy Caddy --> AuthStack["🔑 SSO & Authentication
─────────
Authelia (MFA)
PostgreSQL
Redis"] Caddy --> DataStack["📊 Data Stack
─────────
Airbyte ETL
PostgreSQL
S3 Compat"] Caddy --> SearchStack["🔍 Search Stack
─────────
SearXNG Metasearch
Redis Cache
Elasticsearch"] Caddy --> ObsStack["📈 Observability
─────────
Prometheus
Grafana Dashboards
Loki Logs
AlertManager"] Caddy --> DiagStack["📐 Diagrams
─────────
Mermaid (Self-Hosted)
Architecture Docs"] Caddy --> DBTools["🗄️ DB Tools
─────────
CloudBeaver
PgAdmin
PostgreSQL"] Caddy --> PlatformStack["🚀 Platform
─────────
Dokploy
Container Registry
Deployment"] ObsStack --> Backup["💾 Backup System
Restic Daily Backups
Automated Encryption"] ObsStack --> IDS["🚨 Security Monitoring
Falco Intrusion Detection
Real-time Alerts"] AuthStack --> Vault["🔐 Secrets
Vault + Consul
Secure Secret Storage"] DataStack --> Vault SearchStack --> Vault style Internet fill:#4CAF50,stroke:#333,color:#fff style Caddy fill:#2196F3,stroke:#333,color:#fff style AuthStack fill:#9C27B0,stroke:#333,color:#fff style DataStack fill:#FF9800,stroke:#333,color:#fff style SearchStack fill:#00BCD4,stroke:#333,color:#fff style ObsStack fill:#F44336,stroke:#333,color:#fff style DiagStack fill:#8BC34A,stroke:#333,color:#fff style DBTools fill:#3F51B5,stroke:#333,color:#fff style PlatformStack fill:#E91E63,stroke:#333,color:#fff style Backup fill:#607D8B,stroke:#333,color:#fff style IDS fill:#D32F2F,stroke:#333,color:#fff style Vault fill:#5E35B1,stroke:#333,color:#fff

🔄 Data Flow Architecture

How data moves through the system from ingestion to visualization

graph LR Source["📥 Data Sources
APIs
Databases
External Services"] Source -->|Extract| Airbyte["🔀 Airbyte ETL
Data Integration
Transformation
Scheduling"] Airbyte -->|Load| DataDB["🗄️ PostgreSQL
Data Warehouse
Production Data
Historical Data"] DataDB -->|Query| Analytics["📊 Analytics Engine
Real-time Processing
Aggregations
Reports"] Analytics -->|Metrics| Prometheus["📈 Prometheus
Time-series Database
Metrics Collection
Retention: 7 days"] Prometheus -->|Display| Grafana["📊 Grafana
Visual Dashboards
Alerting
Multi-datasource"] DataDB -->|Logs| Loki["📝 Loki
Log Aggregation
Structured Logs
Query Interface"] Loki -->|Display| Grafana DataDB -->|Cache| Redis["⚡ Redis
Session Cache
Query Results
Rate Limiting"] Source -->|Search| SearXNG["🔍 SearXNG
Metasearch Engine
Multiple Sources
Result Aggregation"] SearXNG -->|Cache| Redis DataDB -->|Backup| Restic["💾 Restic
Daily Backups
Encrypted Storage
Point-in-time Recovery"] DataDB -->|Security| Falco["🚨 Falco
Runtime Monitoring
Threat Detection
Compliance Audit"] Falco -->|Alerts| AlertManager["🔔 AlertManager
Alert Routing
Notification
Escalation"] style Source fill:#4CAF50,stroke:#333,color:#fff style Airbyte fill:#FF9800,stroke:#333,color:#fff style DataDB fill:#3F51B5,stroke:#333,color:#fff style Analytics fill:#F44336,stroke:#333,color:#fff style Prometheus fill:#FF6F00,stroke:#333,color:#fff style Grafana fill:#E91E63,stroke:#333,color:#fff style Loki fill:#00BCD4,stroke:#333,color:#fff style Redis fill:#9C27B0,stroke:#333,color:#fff style SearXNG fill:#00BCD4,stroke:#333,color:#fff style Restic fill:#607D8B,stroke:#333,color:#fff style Falco fill:#D32F2F,stroke:#333,color:#fff style AlertManager fill:#F57C00,stroke:#333,color:#fff

🔒 Security Architecture

Multi-layered security approach protecting the entire infrastructure

graph TB Users["👥 Users"] Users -->|HTTPS| Caddy["🔐 Caddy Edge
TLS 1.3
Let's Encrypt
Security Headers
Rate Limiting"] Caddy -->|Verify| Authelia["🔑 Authelia
TOTP MFA
Session Management
2FA Required"] Authelia -->|Store| Vault["🔐 HashiCorp Vault
Secret Storage
Access Control
Encryption at Rest"] Authelia -->|Access| AppStack["🔒 Protected Services
Microservices
Databases
APIs"] AppStack -->|Monitor| Falco["🚨 Falco Runtime Security
Process Monitoring
File Integrity
Network Anomalies
Threat Detection"] Falco -->|Alert| AlertManager["🔔 Immediate Alerts
Slack Notifications
Email Notifications
Escalation"] AppStack -->|Backup| Restic["💾 Encrypted Backups
AES-256 Encryption
Daily Snapshots
Offsite Storage"] Users -->|Logs| Promtail["📝 Log Collector
Container Logs
System Logs
Application Logs"] Promtail -->|Send| Loki["📝 Loki
Centralized Logs
Query & Search
Audit Trail"] Loki -->|Query| Grafana["📊 Security Dashboard
Failed Auth Attempts
Rate Limit Hits
Falco Alerts"] Vault -->|Manage| KeyRotation["🔄 Key Rotation
Automatic Cycling
Compliance
Audit Logging"] style Users fill:#4CAF50,stroke:#333,color:#fff style Caddy fill:#2196F3,stroke:#333,color:#fff style Authelia fill:#9C27B0,stroke:#333,color:#fff style Vault fill:#5E35B1,stroke:#333,color:#fff style AppStack fill:#F44336,stroke:#333,color:#fff style Falco fill:#D32F2F,stroke:#333,color:#fff style AlertManager fill:#F57C00,stroke:#333,color:#fff style Restic fill:#607D8B,stroke:#333,color:#fff style Promtail fill:#FF6F00,stroke:#333,color:#fff style Loki fill:#00BCD4,stroke:#333,color:#fff style Grafana fill:#E91E63,stroke:#333,color:#fff style KeyRotation fill:#5E35B1,stroke:#333,color:#fff

🐳 Services Topology

Complete mapping of all 27 containerized services

graph TB subgraph EDGE["🔐 EDGE LAYER"] Caddy["Caddy
Reverse Proxy"] Nginx["Nginx
Web Server"] end subgraph AUTH["🔑 AUTHENTICATION & SSO"] Authelia["Authelia
SSO"] AuthDB["PostgreSQL
Auth DB"] end subgraph DATA["📊 DATA STACK"] Airbyte["Airbyte
ETL"] PostgreSQL["PostgreSQL
Data WH"] S3["S3 Storage
Object Storage"] end subgraph SEARCH["🔍 SEARCH STACK"] SearXNG["SearXNG
Metasearch"] Elasticsearch["Elasticsearch
Search Index"] end subgraph OBSERVABILITY["📈 OBSERVABILITY"] Prometheus["Prometheus
Metrics"] Grafana["Grafana
Dashboards"] Loki["Loki
Logs"] AlertManager["AlertManager
Alerts"] NodeExp["Node Exporter
Host Metrics"] cAdvisor["cAdvisor
Container Metrics"] end subgraph SECURITY["🚨 SECURITY & BACKUP"] Falco["Falco
Intrusion Detection"] Restic["Restic
Backups"] Vault["Vault
Secrets"] Consul["Consul
Service Discovery"] end subgraph TOOLS["🛠️ TOOLS & UTILITIES"] CloudBeaver["CloudBeaver
DB Manager"] PgAdmin["PgAdmin
DB Admin"] Mermaid["Mermaid
Diagrams"] Dokploy["Dokploy
Platform"] Terraform["Terraform
IaC"] end subgraph CACHE["⚡ CACHING"] Redis["Redis
Cache/Sessions"] end EDGE --> AUTH EDGE --> DATA EDGE --> SEARCH EDGE --> OBSERVABILITY EDGE --> TOOLS AUTH --> Redis AUTH --> Vault DATA --> Redis DATA --> Vault SEARCH --> Redis OBSERVABILITY --> AlertManager SECURITY --> AlertManager AUTH --> SECURITY DATA --> SECURITY SEARCH --> SECURITY style EDGE fill:#2196F3,stroke:#333,color:#fff style AUTH fill:#9C27B0,stroke:#333,color:#fff style DATA fill:#FF9800,stroke:#333,color:#fff style SEARCH fill:#00BCD4,stroke:#333,color:#fff style OBSERVABILITY fill:#F44336,stroke:#333,color:#fff style SECURITY fill:#D32F2F,stroke:#333,color:#fff style TOOLS fill:#4CAF50,stroke:#333,color:#fff style CACHE fill:#FFD700,stroke:#333,color:#000

🔧 Disaster Recovery & Continuity

Business continuity and disaster recovery procedures

graph TB NormalOps["✅ Normal Operations
All Services Running
Daily Backups
Continuous Monitoring"] IncidentDetect["🚨 Incident Detection
Monitoring Alerts
Health Checks
User Reports"] Assessment["📋 Assessment
Root Cause Analysis
Impact Assessment
RTO/RPO Check"] NormalOps -->|Issue Detected| IncidentDetect IncidentDetect -->|Verify| Assessment Assessment -->|Service Restart| Recovery1["🔄 Service Recovery
Restart Failed Service
Check Dependencies
Verify Health"] Assessment -->|Data Corruption| Recovery2["🗄️ Data Recovery
Restore from Backup
Verify Integrity
Resume Operations"] Assessment -->|Complete Failure| Recovery3["🏗️ Full Rebuild
Deploy Infrastructure
Restore All Data
Failover Complete"] Recovery1 --> Test1["✅ Testing
Service Tests
Integration Tests
Smoke Tests"] Recovery2 --> Test2["✅ Testing
Data Validation
Application Tests"] Recovery3 --> Test3["✅ Testing
Full Validation
Load Testing"] Test1 -->|Pass| NormalOps Test2 -->|Pass| NormalOps Test3 -->|Pass| NormalOps Test1 -->|Fail| Escalate["📞 Escalation
Manual Intervention
Engineering Team"] Test2 -->|Fail| Escalate Test3 -->|Fail| Escalate Escalate -->|Resolved| NormalOps NormalOps -->|Periodic| DRTest["🧪 DR Testing
Monthly Backups Test
Failover Drills
Documentation Update"] DRTest --> NormalOps style NormalOps fill:#4CAF50,stroke:#333,color:#fff style IncidentDetect fill:#F57C00,stroke:#333,color:#fff style Assessment fill:#FF9800,stroke:#333,color:#fff style Recovery1 fill:#2196F3,stroke:#333,color:#fff style Recovery2 fill:#00BCD4,stroke:#333,color:#fff style Recovery3 fill:#9C27B0,stroke:#333,color:#fff style Test1 fill:#4CAF50,stroke:#333,color:#fff style Test2 fill:#4CAF50,stroke:#333,color:#fff style Test3 fill:#4CAF50,stroke:#333,color:#fff style Escalate fill:#D32F2F,stroke:#333,color:#fff style DRTest fill:#5E35B1,stroke:#333,color:#fff

🚀 Deployment Pipeline & Updates

Continuous deployment and infrastructure updates workflow

graph LR Dev["👨💻 Developer
Code Push"] Dev -->|Git Push| Repo["📦 Git Repository
Main Branch
Docker Images"] Repo -->|Webhook| Build["🔨 Build Pipeline
Docker Build
Image Scan
Security Check"] Build -->|Success| Registry["📦 Docker Registry
Container Images
Version Tags"] Build -->|Fail| Notify1["📧 Notify Dev
Build Failed
Logs Available"] Notify1 -->|Fix & Push| Repo Registry -->|Deploy| Dev1["🧪 Dev Environment
Test Deployment
Smoke Tests
Integration Tests"] Dev1 -->|Pass| Staging["🔄 Staging Environment
Full Testing
Performance Tests
Security Scan"] Dev1 -->|Fail| Notify2["⚠️ Build Issue
Testing Failed
Rollback Container"] Staging -->|Pass| Approval["✅ Approval Gate
Manual Review
Change Log
Rollback Plan"] Staging -->|Fail| Notify2 Approval -->|Approved| Prod["🚀 Production
Blue-Green Deploy
Health Checks
Gradual Rollout"] Approval -->|Rejected| Cancel["❌ Deployment Cancelled
Continue Testing
Fix Issues"] Cancel -->|Ready| Approval Prod -->|Success| Monitor["📊 Monitor
Metrics
Logs
Alerts"] Prod -->|Failure| Rollback["⏮️ Rollback
Previous Version
Incident Report
Postmortem"] Rollback -->|Analyze| Repo Monitor -->|Healthy| NormalOps["✅ Normal Operations
Continuous Monitoring"] style Dev fill:#4CAF50,stroke:#333,color:#fff style Repo fill:#2196F3,stroke:#333,color:#fff style Build fill:#FF9800,stroke:#333,color:#fff style Registry fill:#00BCD4,stroke:#333,color:#fff style Dev1 fill:#9C27B0,stroke:#333,color:#fff style Staging fill:#E91E63,stroke:#333,color:#fff style Approval fill:#5E35B1,stroke:#333,color:#fff style Prod fill:#2196F3,stroke:#333,color:#fff style Monitor fill:#F44336,stroke:#333,color:#fff style Rollback fill:#D32F2F,stroke:#333,color:#fff style NormalOps fill:#4CAF50,stroke:#333,color:#fff style Notify1 fill:#F57C00,stroke:#333,color:#fff style Notify2 fill:#F57C00,stroke:#333,color:#fff style Cancel fill:#FF6F00,stroke:#333,color:#fff