Terraform Installation ====================== This guide covers deploying OneByZero Neo on enterprise AWS accounts using Terraform and Helm for infrastructure provisioning and application deployment. Overview -------- OneByZero Neo is a modular AI platform supporting Chatbot, Voice Bot, and Agentic Flow agents. The system runs in a multi-zone Amazon EKS cluster for high availability and uses GitOps (ArgoCD) for continuous deployment. The platform integrates with: - **Amazon RDS PostgreSQL** - Primary relational database - **Amazon OpenSearch** - Vector storage and semantic search - **ClickHouse** - Time-series analytics database - **Amazon S3** - Object storage for documents and models Infrastructure as a Service Deployment -------------------------------------- OneByZero Neo follows an Infrastructure as a Service (IaaS) deployment model on AWS, where the complete platform is deployed within the enterprise's own AWS account. This approach provides: **Data Sovereignty** All data remains within the enterprise's AWS account and VPC. No data is transmitted to external systems, ensuring compliance with data residency requirements. **Network Isolation** The platform operates within private subnets with configurable security groups and network ACLs. External access is controlled through AWS Application Load Balancer with SSL termination. **IAM Integration** Services authenticate to AWS resources using IAM Roles for Service Accounts (IRSA), eliminating the need for long-lived credentials. **Resource Control** Enterprises maintain full control over compute resources, scaling policies, and cost management through AWS native tools. Deployment Architecture ~~~~~~~~~~~~~~~~~~~~~~~ .. figure:: /images/deployment-architecture.png :alt: OneByZero Neo AWS Deployment Architecture :align: center :width: 100% OneByZero Neo deployment architecture showing the Enterprise AWS Account with VPC, EKS cluster node pools, and AWS managed services. Prerequisites ------------- Infrastructure Requirements ~~~~~~~~~~~~~~~~~~~~~~~~~~~ - **Amazon EKS cluster** (v1.24+) - **Node groups** with appropriate taints and labels: - ``app-node`` pool for application workloads - ``studiops-node`` pool for Langfuse/ClickHouse - ``rayservice-head``, ``rayservice-gpu-worker``, ``rayservice-cpu-worker`` for Whisper service - **AWS Application Load Balancer Controller** - **External DNS Controller** - **Amazon RDS PostgreSQL** instance - **Amazon OpenSearch** cluster - **AWS S3 buckets** for data storage - **AWS SQS queues** for message processing Required AWS Resources ~~~~~~~~~~~~~~~~~~~~~~ **RDS Database** PostgreSQL instance for application data persistence. **OpenSearch Domain** Managed OpenSearch cluster for vector storage and semantic search. **S3 Buckets** - Data ingestion bucket - StudioOps bucket - StudioOps datasets bucket - Bedrock knowledge base data ingestion bucket - Supplemental storage bucket - Whisper models bucket - Voice schedule bucket - Bedrock evaluators bucket **SQS Queues** - KPI events queue - Voice calls queue Security Requirements ~~~~~~~~~~~~~~~~~~~~~ - **IAM Role** for EKS IRSA (IAM Roles for Service Accounts) - **SSL Certificate** from AWS Certificate Manager (ACM) - **Application Load Balancer** for ingress traffic Password Generation Guidelines ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Database passwords (RDS PostgreSQL, Redis, ClickHouse) should use URL-safe characters to prevent connection string parsing issues. **Safe Characters (No Encoding Required)** .. code-block:: text A-Z a-z 0-9 _ - . ~ **Characters to Avoid** .. code-block:: text @ : / ? # & = % + (space) ; " ' < > \ ^ ` { } [ ] If special characters must be used, URL-encode the password before storing: .. code-block:: bash python3 -c "import urllib.parse; print(urllib.parse.quote_plus('your-password'))" Keys Generation ~~~~~~~~~~~~~~~ Generate required encryption keys and secrets before deployment: .. list-table:: :header-rows: 1 :widths: 30 30 40 * - Service - Variable - Generation Command * - Langfuse - SALT - ``openssl rand -base64 32`` * - Langfuse - ENCRYPTION_KEY - ``openssl rand -hex 32`` * - Langfuse - LANGFUSE_KEY - ``openssl rand -base64 32`` * - Langfuse - NEXTAUTH_SECRET - ``openssl rand -base64 32`` * - Agentic Flow - DEFAULT_SUPERUSER_PASSWORD - ``openssl rand -base64 32`` * - Agentic Flow - LANGFLOW_SECRET_KEY - ``openssl rand -base64 32`` * - Bot Builder - ENCRYPTION_SECRET - ``openssl rand -base64 24`` * - Bot Builder - JWT_SIGNING_KEY_ID - ``openssl rand -base64 24`` * - Knowledge Base - PE_PASSPHRASE - ``openssl rand -base64 43`` **JWT Key Pair Generation** .. code-block:: bash # Generate private key openssl genpkey -algorithm RSA -out private_key.pem # Generate public key from private key openssl rsa -pubout -in private_key.pem -out public_key.pem Helm Deployment Process ----------------------- Pre-deployment Validation ~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash # Verify cluster connectivity kubectl cluster-info # Check node readiness kubectl get nodes # Verify required namespaces kubectl create namespace --dry-run=client -o yaml | kubectl apply -f - # Validate AWS Load Balancer Controller kubectl get deployment -n kube-system aws-load-balancer-controller Deploy Infrastructure Services ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Deploy infrastructure services first as they are dependencies for application services. **Voice Redis** .. code-block:: bash helm upgrade --install voice-redis ./voice_redis \ --namespace \ --create-namespace \ --wait \ --timeout 10m **Langfuse (StudioOps)** .. code-block:: bash helm upgrade --install langfuse ./langfuse \ --namespace \ --wait \ --timeout 15m **Hulaho (Analytics Dashboard)** .. code-block:: bash helm upgrade --install hulaho ./hulaho \ --namespace \ --wait \ --timeout 15m Deploy Core Application Services ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Deploy services in dependency order: .. code-block:: bash # EKS Jobs Managing Service (required by Knowledge Base) helm upgrade --install eks-jobs-managing ./eks_jobs_managing_service \ --namespace \ --wait \ --timeout 10m # Knowledge Base Service helm upgrade --install knowledge-base ./knowledge_base \ --namespace \ --wait \ --timeout 10m # Agentic Flow Service helm upgrade --install agentic-flow ./agentic_flow \ --namespace \ --wait \ --timeout 10m # Bot Builder Services helm upgrade --install botbuilder ./botbuilder \ --namespace \ --wait \ --timeout 10m # KPI Events Polling Service helm upgrade --install kpi-events-polling ./kpi_events_polling_service \ --namespace \ --wait \ --timeout 10m # Threads Service helm upgrade --install threads-service ./threads_service \ --namespace \ --wait \ --timeout 10m Deploy Voice Services ~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash # Voice Backend helm upgrade --install voice-backend ./voice_backend \ --namespace \ --wait \ --timeout 10m # Voice Agent helm upgrade --install voice-agent ./voice_agent \ --namespace \ --wait \ --timeout 10m # Voice Scheduler helm upgrade --install voice-scheduler ./voice_scheduler \ --namespace \ --wait \ --timeout 10m # Whisper Service (Ray-based) helm upgrade --install whisper ./whisper \ --namespace \ --wait \ --timeout 15m Verification ~~~~~~~~~~~~ .. code-block:: bash # Check all deployments kubectl get deployments -n # Check all services kubectl get services -n # Check ingress resources kubectl get ingress -n # Check pod status kubectl get pods -n # View logs for troubleshooting kubectl logs -f deployment/ -n Upgrade and Rollback ~~~~~~~~~~~~~~~~~~~~ **Upgrade Process** .. code-block:: bash helm upgrade ./ \ --namespace \ --reuse-values \ --wait \ --timeout 10m **Rollback Process** .. code-block:: bash # List release history helm history --namespace # Rollback to previous version helm rollback --namespace # Rollback to specific revision helm rollback --namespace Service Configuration Reference ------------------------------- This section documents all required environment variables for each microservice. Variables not listed are considered optional and use default values. .. note:: All secrets should be stored in HashiCorp Vault or Kubernetes Secrets and mapped to deployments using the VaultStaticSecrets CRD or native Kubernetes secret references. Agentic Flow Service ~~~~~~~~~~~~~~~~~~~~ AI workflow orchestration and LangFlow integration service that manages AI workflows and provides API endpoints for conversational AI applications. **Network & Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ADDITIONAL_CORS_ORIGINS - Comma-separated list of allowed CORS origin domains * - HOST - Application bind address for container networking * - APPLICATION_PORT - Port number for the application server **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LANGFLOW_DATABASE_URL - PostgreSQL connection string for workflow persistence * - LANGFLOW_SCHEMA_NAMES - Database schemas used by LangFlow for multi-tenant data organization **Logging & Monitoring** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LANGFLOW_LOG_LEVEL - Application logging verbosity level * - STUDIOTELEMETRY_COLLECTOR_ENDPOINT - OpenTelemetry metrics collection endpoint * - STUDIOTELEMETRY_COLLECTOR_AUTH_TOKEN - Authentication token for telemetry data submission **API Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - GUARDRAIL_API_BASE_URL - API endpoint for content guardrails and safety checks **OpenSearch Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - INHOUSE_OPENSEARCH_HOST - OpenSearch cluster endpoint for vector storage * - INHOUSE_OPENSEARCH_USERNAME - OpenSearch authentication username * - INHOUSE_OPENSEARCH_PASSWORD - OpenSearch authentication password **AWS Bedrock Knowledge Base Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - BEDROCK_KNOWLEDGE_BASE_OPENSEARCH_HOST - Bedrock knowledge base OpenSearch endpoint * - BEDROCK_KNOWLEDGE_BASE_OPENSEARCH_USERNAME - Bedrock knowledge base OpenSearch username * - BEDROCK_KNOWLEDGE_BASE_OPENSEARCH_PASSWORD - Bedrock knowledge base OpenSearch password **Resource Requirements** - CPU: 1000m (requests) / 1290m (limits) - Memory: 1Gi (requests) / 2Gi (limits) **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/health_check`` **Dependencies**: PostgreSQL, OpenSearch, AWS Bedrock, StudioOps telemetry service Bot Builder Service ~~~~~~~~~~~~~~~~~~~ Conversational bot creation and management platform with separate builder and viewer components for designing and running chatbots. **Builder Component - Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - APPLICATION_HOSTNAME - Application bind address * - APPLICATION_PORT - Application server port * - BASE_URL_FROM_ENV - Base URL for the builder application **Authentication & Security** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AZURE_AD_CLIENT_ID - Azure Active Directory application client ID * - AZURE_AD_CLIENT_SECRET - Azure AD application secret * - AZURE_AD_TENANT_ID - Azure AD tenant identifier * - NEXTAUTH_URL - NextAuth authentication service URL * - JWT_SIGNING_KEY_ID - JWT token signing key identifier * - JWT_EXPIRATION_IN_HOURS - JWT token expiration time in hours * - ENCRYPTION_SECRET - Data encryption secret key * - JWT_PRIVATE_KEY_PATH - Path to JWT private key PEM file (mounted from secrets) * - JWT_PUBLIC_KEY_PATH - Path to JWT public key PEM file (mounted from secrets) **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string for bot configurations **Bot Behavior Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - BUILDER_ALLOWED_ORIGINS - CORS allowed origins for development * - BUTTON_CHOICE_INTERRUPTION - Enable button choice interruption in conversations * - FILE_INPUT_INTERRUPTION - Enable file input interruption handling * - PICTURE_CHOICE_INTERRUPTION - Enable picture choice interruption handling * - CHAT_HISTORY_LIMIT - Maximum number of chat history entries to retain **External Service Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DIRECTLINE_SERVICE_BASE_URL - Microsoft Bot Framework DirectLine API endpoint * - NEXT_PUBLIC_GOOGLE_API_KEY - Google Maps/Services API key **Organization & User Management** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ORGANIZATION_SETUP_AUTH_TOKEN - Organization setup authentication token (must match LANGFUSE_KEY) * - ORGANIZATION_SETUP_EMAIL - Default organization setup email * - DEFAULT_WORKSPACE_PLAN - Default workspace plan for new users * - ADMIN_EMAIL - System administrator email address * - SEED_DATA - Whether to seed initial data on startup **Session Management** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - NEXT_PUBLIC_SESSION_EXPIRY_TIMEOUT_IN_HOURS - Session timeout in hours for security **Metrics & Analytics** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - METRICS_API_URL - Metrics collection API endpoint * - METRICS_API_AUTH_TOKEN - Authentication token for metrics API * - HULAHO_API_BASE_URL - Hulaho (Superset) analytics endpoint * - HULAHO_ADMIN_PASSWORD - Hulaho user authentication password **AWS S3 Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - S3_ENDPOINT - S3 service endpoint * - S3_BUCKET - Primary S3 bucket for file storage * - S3_REGION - AWS region for S3 operations * - IS_IRSA_REQUIRED - Enable IAM Roles for Service Accounts * - DATASET_S3_BUCKET - S3 bucket for dataset storage * - DATASET_S3_REGION - AWS region for dataset bucket * - DATASET_S3_FORCE_PATH_STYLE - Force path-style S3 URLs **Service Integration URLs** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - NEXT_PUBLIC_VIEWER_URL - Bot viewer application URL * - NEXT_PUBLIC_VOICE_AGENT_BACKEND_SERVICE_BASE_URL - Voice backend service URL * - NEXT_PUBLIC_VOICE_AGENT_BACKEND_WS_URL - Voice backend WebSocket URL * - NEXT_PUBLIC_AGENTIC_FLOW_URL - Agentic flow service URL * - NEXT_PUBLIC_KNOWLEDGE_BASE_URL - Knowledge base service URL * - NEXT_PUBLIC_DATASET_BASE_URL - Dataset management URL **Analytics Dashboard Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - NEXT_PUBLIC_HULAHO_BASE_URL - Analytics dashboard base URL * - NEXT_PUBLIC_HULAHO_VOICE_DASHBOARD_ID - Voice analytics dashboard identifier * - NEXT_PUBLIC_HULAHO_CHAT_DASHBOARD_ID - Chat analytics dashboard identifier * - NEXT_PUBLIC_HULAHO_FLOW_DASHBOARD_ID - Flow analytics dashboard identifier **Feature Flags** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - NEXT_PUBLIC_IS_VOICE_FEATURE_ENABLED - Enable voice bot features * - NEXT_PUBLIC_IS_CHAT_FEATURE_ENABLED - Enable chat bot features * - NEXT_PUBLIC_IS_FLOW_FEATURE_ENABLED - Enable flow-based bot features * - NEXT_PUBLIC_DEMO_MODE - Enable demo mode features **Webhook Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - WEBHOOK_TASK_EXPIRATION_MINUTES - Webhook task timeout in minutes **Branding** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - NEXT_PUBLIC_COPYWRITE_NAME - Copyright notice text * - NEXT_PUBLIC_AWS_MODEL_REGION - AWS region for AI model access **Viewer Component - Additional Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AWS_SQS_QUEUE_URL - SQS queue URL for KPI events * - AWS_SQS_QUEUE_REGION - AWS region for SQS operations * - LLM_HOST_URLS - Large Language Model service endpoints * - DATASET_S3_ENDPOINT - S3 endpoint for dataset operations **Resource Requirements (Viewer)** - CPU: 1200m (requests) / 1290m (limits) - Memory: 1Gi (requests) / 2Gi (limits) **Autoscaling (Viewer)** - Min Replicas: 1 - Max Replicas: 2 - Target CPU Utilization: 80% **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Dependencies**: PostgreSQL, S3, Azure AD, Voice services, Agentic Flow, Knowledge Base Knowledge Base Service ~~~~~~~~~~~~~~~~~~~~~~ Document ingestion, processing, and retrieval service that manages knowledge bases, handles document uploads, and provides semantic search capabilities. **Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - APPLICATION_NAME - Service identifier for logging and monitoring * - PORT - Application server port * - PE_PASSPHRASE - Private encryption passphrase for sensitive data **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string for metadata storage **Service Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - EKS_JOBS_MANAGING_SERVICE_BASE_URL - EKS job orchestration service endpoint **AWS Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - REGION_NAME - Primary AWS region for all operations * - DATA_INGESTION_ROLE_ARN - IAM role ARN for data ingestion operations **S3 Storage Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - SOURCE_FILES_S3_BUCKET_NAME - S3 bucket for source document storage * - SOURCE_FILES_S3_BUCKET_REGION - AWS region for source files bucket * - MARKDOWN_FILES_S3_BUCKET_NAME - S3 bucket for processed markdown files * - MARKDOWN_FILES_S3_BUCKET_REGION - AWS region for markdown files bucket **OpenSearch Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - OPENSEARCH_HOST - OpenSearch cluster endpoint for vector storage * - OPENSEARCH_USERNAME - OpenSearch authentication username * - OPENSEARCH_PASSWORD - OpenSearch authentication password * - OPENSEARCH_DOMAIN_ARN - OpenSearch domain ARN * - OPENSEARCH_AWS_REGION - AWS region for OpenSearch operations **Embedding Model Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - EMBEDDING_MODEL_AWS_REGION - AWS region for embedding model * - EMBEDDING_MODEL_ID - AWS Bedrock embedding model identifier **Data Ingestion Service Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATA_INGESTION_APPLICATION_NAME - Data ingestion service identifier * - DATA_INGESTION_S3_BUCKET_NAME - S3 bucket for ingestion pipeline * - DATA_INGESTION_JOB_CONTAINER_NAME - Kubernetes job container name * - DATA_INGESTION_JOB_NAMESPACE - Kubernetes namespace for ingestion jobs * - DATA_INGESTION_JOB_COMMAND - Job execution command * - DATA_INGESTION_JOB_ECR_IMAGE_URI - Container image URI for ingestion jobs **AI Model Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATA_INGESTION_ANTHROPIC_API_KEY - Anthropic Claude API key (if using Anthropic directly) **AWS Bedrock Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATA_INGESTION_AWS_BEDROCK_REGION_NAME - AWS region for Bedrock operations * - DATA_INGESTION_AWS_BEDROCK_MODEL_NAME - Bedrock model identifier * - DATA_INGESTION_AWS_BEDROCK_MODEL_MAX_TOKENS - Maximum tokens per request * - DATA_INGESTION_AWS_BEDROCK_MODEL_TEMPERATURE - Model temperature for response variability * - DATA_INGESTION_AWS_BEDROCK_MODEL_TOP_P - Top-p sampling parameter * - DATA_INGESTION_AWS_BEDROCK_MODEL_TOP_K - Top-k sampling parameter **Bedrock Knowledge Base Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - BEDROCK_KNOWLEDGE_BASE_ROLE_ARN - IAM role for Bedrock knowledge base access * - BEDROCK_KNOWLEDGE_BASE_REGION - AWS region for Bedrock knowledge base * - BEDROCK_KNOWLEDGE_BASE_EMBEDDING_MODEL_ARN - Embedding model ARN * - BEDROCK_KNOWLEDGE_BASE_PARSING_MODEL_ARN - Document parsing model ARN * - BEDROCK_KNOWLEDGE_BASE_SUPPLEMENTAL_S3_URI - Supplemental storage S3 URI * - BEDROCK_KNOWLEDGE_BASE_DATA_SOURCE_S3_ARN - Data source S3 bucket ARN * - BEDROCK_KNOWLEDGE_BASE_DATA_SOURCE_S3_OWNER_ACCOUNT_ID - AWS account ID for S3 bucket ownership * - BEDROCK_KNOWLEDGE_BASE_DATA_SOURCE_S3_BUCKET_NAME - S3 bucket for Bedrock data source * - BEDROCK_KNOWLEDGE_BASE_DATA_SOURCE_S3_BUCKET_REGION - AWS region for Bedrock data source bucket **Bedrock Knowledge Base OpenSearch Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - BEDROCK_KNOWLEDGE_BASE_OPENSEARCH_HOST - OpenSearch endpoint for Bedrock KB * - BEDROCK_KNOWLEDGE_BASE_OPENSEARCH_DOMAIN_ARN - OpenSearch domain ARN for Bedrock KB * - BEDROCK_KNOWLEDGE_BASE_OPENSEARCH_USERNAME - OpenSearch username for Bedrock KB * - BEDROCK_KNOWLEDGE_BASE_OPENSEARCH_PASSWORD - OpenSearch password for Bedrock KB **Resource Requirements** - CPU: 1200m (requests) / 1290m (limits) - Memory: 1Gi (requests) / 2Gi (limits) **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/api/v1/health`` **Dependencies**: OpenSearch, S3, AWS Bedrock, EKS Jobs Managing Service, PostgreSQL Voice Agent Service ~~~~~~~~~~~~~~~~~~~ Real-time voice conversation handling service that manages voice calls, integrates with speech services, and provides conversational AI capabilities. **Agent Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AGENT_NAME - AI agent implementation type * - APPLICATION_NAME - Service identifier for logging and monitoring * - BASE_URL - Base URL for the voice agent service **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string for conversation persistence **AWS Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AWS_SQS_QUEUE_URL - SQS queue URL for KPI events and analytics * - AWS_REGION_NAME - AWS region for all AWS service operations **Azure OpenAI Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AZURE_OPENAI_API_KEY - Azure OpenAI service API key * - AZURE_OPENAI_API_VERSION - Azure OpenAI API version * - AZURE_OPENAI_DEPLOYMENT - Azure OpenAI deployment name * - AZURE_OPENAI_ENDPOINT - Azure OpenAI service endpoint * - AZURE_OPENAI_MODEL_NAME - Azure OpenAI model identifier **Azure Speech Services Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AZURE_SPEECH_KEY - Azure Speech Services API key * - AZURE_SPEECH_REGION - Azure Speech Services region * - AZURE_CANDIDATE_LANGUAGES - Supported languages for speech recognition (comma-separated) **ElevenLabs Voice Synthesis Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ELEVENLABS_API_KEY - ElevenLabs API key for voice synthesis * - ELEVENLABS_EXPERIMENTAL_STREAMING - Enable experimental streaming features * - ELEVENLABS_EXPERIMENTAL_WEBSOCKET - Enable WebSocket-based streaming * - ELEVENLABS_MODEL_ID - ElevenLabs voice model identifier * - ELEVENLABS_OPTIMIZE_STREAMING_LATENCY - Latency optimization level (1-4) * - ELEVENLABS_VOICE_ID - Specific voice ID for synthesis * - ELEVENLABS_VOICE_SPEED - Voice playback speed multiplier * - ELEVENLABS_VOICE_STYLE - Voice style variation parameter **Twilio Telephony Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - TWILIO_ACCOUNT_SID - Twilio account identifier * - TWILIO_AUTH_TOKEN - Twilio authentication token * - TWILIO_EDGE - Twilio edge location for optimal routing * - TWILIO_REGION - Twilio region for service operations * - FROM_PHONE_NUMBER - Outbound caller ID phone number **Redis Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - REDISHOST - Redis server hostname for session storage * - REDISPORT - Redis server port * - REDISPASSWORD - Redis authentication password * - REDISDB - Redis database number **Conversation Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ENABLE_FILLERS - Enable/disable filler audio during processing delays * - FILLER_AUDIO_PATH - Path to filler audio files **Caching Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LANGCHAIN_CACHE - LangChain caching strategy for improved performance **Logging Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LOG_LEVEL - Application logging verbosity level **Telemetry Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - STUDIOTELEMETRY_COLLECTOR_ENDPOINT - OpenTelemetry collector endpoint * - STUDIOTELEMETRY_COLLECTOR_AUTH_TOKEN - Authentication token for telemetry * - STUDIOTELEMETRY_APP_NAME - Application name for telemetry identification **Conversation Schema Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - TRANSCRIPT_SUMMARY_SCHEMA - JSON schema defining conversation summary structure **Resource Requirements** - CPU: 250m (requests) / 500m (limits) - Memory: 512Mi (requests) / 3Gi (limits) **Autoscaling** - Min Replicas: 1 - Max Replicas: 1 - Target CPU Utilization: 70% - Target Memory Utilization: 80% **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/`` **Dependencies**: Redis, PostgreSQL, Azure OpenAI, Azure Speech, ElevenLabs, Twilio, SQS Voice Backend Service ~~~~~~~~~~~~~~~~~~~~~ Voice call management and scheduling backend service that handles voice call orchestration, scheduling, and integration with telephony services. **AWS Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AWS_SQS_QUEUE_URL - SQS queue URL for voice call processing * - AWS_REGION_NAME - AWS region for all AWS service operations **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string for call data persistence **ElevenLabs Voice Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ELEVENLABS_API_KEY - ElevenLabs API key for voice synthesis * - ELEVENLABS_EXPERIMENTAL_STREAMING - Enable experimental streaming features * - ELEVENLABS_EXPERIMENTAL_WEBSOCKET - Enable WebSocket-based streaming * - ELEVENLABS_MODEL_ID - ElevenLabs voice model identifier * - ELEVENLABS_OPTIMIZE_STREAMING_LATENCY - Latency optimization level * - ELEVENLABS_VOICE_ID - Specific voice ID for synthesis **Twilio Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - TWILIO_ACCOUNT_SID - Twilio account identifier * - TWILIO_AUTH_TOKEN - Twilio authentication token * - TWILIO_EDGE - Twilio edge location for routing * - TWILIO_REGION - Twilio region for operations * - FROM_PHONE_NUMBER - Outbound caller ID phone number **Azure OpenAI Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AZURE_API_KEY - Azure OpenAI API key * - AZURE_ENDPOINT - Azure OpenAI endpoint **Analytics Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - HULAHO_API_BASE_URL - Analytics API endpoint * - HULAHO_ADMIN_USERNAME - Analytics platform admin username * - HULAHO_ADMIN_PASSWORD - Analytics platform admin password **Scheduling Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - SCHEDULE_BUCKET_NAME - S3 bucket for call scheduling data * - SCHEDULER_LAMBDA_DURATION_IN_MILLIS - Lambda function timeout in milliseconds **Service Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ENABLE_CONTACT_POLICY - Enable/disable contact policy enforcement * - LOG_LEVEL - Application logging verbosity * - QUEUE_SERVICE - Message queue service type **Resource Requirements** - CPU: 1000m (requests) / 1290m (limits) - Memory: 1Gi (requests) / 2Gi (limits) **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/health`` **Dependencies**: SQS, S3, ElevenLabs, Twilio, PostgreSQL, Azure OpenAI Voice Scheduler Service ~~~~~~~~~~~~~~~~~~~~~~~ Automated voice call scheduling and management service that handles call timing, contact policies, and queue management. **Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - APPLICATION_NAME - Service identifier for logging and monitoring **Scheduling Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ALLOWED_WEEKDAYS - Comma-separated list of allowed days for scheduling calls (0=Monday) * - TIMEZONE - Timezone for scheduling operations * - SUCCESSFUL_CALLS_ALLOWED - Maximum successful calls per contact * - RETRIES_ALLOWED - Maximum retry attempts per call **Contact Policy Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - ENABLE_CONTACT_POLICY - Enable/disable contact policy enforcement * - CONTACT_POLICY_ENABLED - Global contact policy toggle * - GLOBAL_CONTACT_POLICY_START_TIME_IN_24_HOURS_FORMAT - Daily contact window start time * - GLOBAL_CONTACT_POLICY_END_TIME_IN_24_HOURS_FORMAT - Daily contact window end time **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string for scheduling data **Azure Service Bus Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AZURE_BUS_CONNECTION_STRING - Azure Service Bus connection string * - AZURE_BUS_QUEUE_NAME - Azure Service Bus queue name **AWS SQS Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AWS_SQS_QUEUE_URL - SQS queue URL for voice call processing * - AWS_REGION_NAME - AWS region for SQS operations **Service Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - VOICE_SERVICE_BASE_URL - Voice agent service internal URL * - QUEUE_SERVICE - Primary queue service type (azure/aws) **Logging Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LOG_LEVEL - Application logging verbosity level **Resource Requirements** - CPU: 1000m (requests) / 1290m (limits) - Memory: 1Gi (requests) / 2Gi (limits) **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/health`` **Dependencies**: Azure Service Bus, SQS, PostgreSQL, Voice Agent Service Whisper Service (Ray-based) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Distributed speech-to-text transcription service using Ray framework for scalable audio processing with GPU acceleration. **Hugging Face Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - hf_token - Hugging Face API token for model access **Ray Service Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Configuration - Description * - Image - Container image URI for Whisper Ray service * - Image Pull Policy - Always pull latest image **Head Node Configuration** - Resources: 1-2 CPU, 4-10Gi memory - Node Selector: ``node-pool: rayservice-head`` - Tolerations: ``ray.io/node-type=head:NoSchedule`` **GPU Worker Configuration** - Enabled: true - Replicas: 1 (min: 1, max: 20) - Resources: 2-3 CPU, 6-12Gi memory, 1 GPU - Node Selector: ``node-pool: rayservice-gpu-worker`` - Tolerations: ``ray.io/node-type=gpu-worker:NoSchedule`` **CPU Worker Configuration** - Enabled: true - Replicas: 1 (min: 1, max: 20) - Resources: 2-3 CPU, 6-12Gi memory - Node Selector: ``node-pool: rayservice-cpu-worker`` - Tolerations: ``ray.io/node-type=cpu-worker:NoSchedule`` **Serve Configuration** - Working Directory: S3 URL to model package - Deployment Services: - **TranscriptionServer**: Max 100 concurrent queries, autoscaling 1-20 replicas - **FasterWhisperASR**: Max 10 concurrent queries, 32GB memory, autoscaling 1-20 replicas - **SileroVAD**: Voice Activity Detection, max 5 concurrent queries, autoscaling 1-20 replicas **Ingress Configuration** - Whisper Dashboard: Port 8265 - Whisper Serve API: Port 8000 **Dependencies**: Ray cluster, GPU nodes, S3 for model storage, Hugging Face KPI Events Polling Service ~~~~~~~~~~~~~~~~~~~~~~~~~~ Analytics and metrics collection service that processes platform events and generates insights using AI models. **Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - APPLICATION_NAME - Service identifier * - APPLICATION_VERSION - Application version for tracking * - RUNTIME_MODE - Service operation mode * - PORT - Application server port **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string for event storage **Queue Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - QUEUE_SERVICE - Queue service provider * - KPI_EVENTS_SQS_QUEUE_NAME - SQS queue name for KPI events * - KPI_EVENTS_SQS_REGION_NAME - AWS region for KPI events queue * - AWS_SQS_QUEUE_NAME - AWS SQS queue name * - AWS_REGION_NAME - AWS region for operations **AWS Bedrock Model Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AWS_BEDROCK_MODEL_NAME - Bedrock model identifier for event analysis * - AWS_BEDROCK_MODEL_MAX_TOKENS - Maximum tokens per request * - AWS_BEDROCK_MODEL_TEMPERATURE - Model temperature for deterministic responses * - AWS_BEDROCK_MODEL_TOP_K - Top-k sampling parameter * - AWS_BEDROCK_MODEL_TOP_P - Top-p sampling parameter **Telemetry Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - STUDIOOPS_OTEL_ENDPOINT - OpenTelemetry endpoint for metrics **Logging Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LOG_LEVEL - Application logging verbosity level **Resource Requirements** - CPU: 200m (requests) / 1290m (limits) - Memory: 512Mi (requests) / 2Gi (limits) **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/api/v1/health`` **Dependencies**: SQS, AWS Bedrock, PostgreSQL, StudioOps EKS Jobs Managing Service ~~~~~~~~~~~~~~~~~~~~~~~~~ Kubernetes job orchestration and management service that handles batch processing and job lifecycle management. **Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - APPLICATION_NAME - Service identifier for logging and monitoring * - PORT - Application server port **Resource Requirements**: Default allocation **Node Placement**: Default (no specific node selector configured) **Health Check**: ``/api/v1/health`` **Dependencies**: Kubernetes API, RBAC permissions for job management Voice Redis Service ~~~~~~~~~~~~~~~~~~~ In-memory data store for voice services session management and caching. **Redis Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Configuration - Description * - Architecture - Deployment architecture (standalone/cluster) * - Authentication - Password authentication enabled * - Password - Redis authentication password * - Persistence - Data persistence configuration (disabled for performance) **Resource Requirements** - CPU: 1200m (requests) / 1290m (limits) - Memory: 1Gi (requests) / 2Gi (limits) **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: Redis ping command **Dependencies**: None (standalone service) Langfuse (StudioOps) Service ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ LLM observability, analytics, and evaluation platform providing comprehensive monitoring and analysis of AI model performance. **Components** - **Web Interface**: Dashboard and API endpoints - **Worker**: Background processing for analytics - **ClickHouse**: Time-series database for metrics storage - **PostgreSQL**: Metadata and configuration storage **Core Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LANGFUSE_KEY - Langfuse API key for authentication * - NEXTAUTH_URL - NextAuth authentication URL * - NEXT_PUBLIC_LANGFUSE_CLOUD_REGION - Cloud region (empty for self-hosted) * - NODE_OPTIONS - Node.js memory optimization flags **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string * - DIRECT_URL - Direct database connection URL **ClickHouse Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - CLICKHOUSE_MIGRATION_SSL - SSL for ClickHouse migrations * - CLICKHOUSE_CLUSTER_ENABLED - Cluster deployment mode * - LANGFUSE_AUTO_CLICKHOUSE_MIGRATION_DISABLED - Automatic migration toggle * - LANGFUSE_CLICKHOUSE_ENABLED - ClickHouse integration toggle * - CLICKHOUSE_DB - ClickHouse database name **Feature Flags** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES - Enable experimental features * - LANGFUSE_ENABLE_BACKGROUND_MIGRATIONS - Enable background database migrations **AWS S3 Dataset Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LANGFUSE_S3_DATASET_UPLOAD_BUCKET - S3 bucket for dataset uploads * - LANGFUSE_S3_DATASET_UPLOAD_PREFIX - S3 prefix for dataset files * - LANGFUSE_S3_DATASET_UPLOAD_REGION - AWS region for S3 operations * - LANGFUSE_S3_DATASET_UPLOAD_ENDPOINT - S3 endpoint URL * - LANGFUSE_S3_DATASET_UPLOAD_FORCE_PATH_STYLE - Force path-style S3 URLs * - AWS_REGION - AWS region for all operations **Bedrock Evaluation Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - EKS_JOB_MANAGING_SERVICE_TRIGGER_URL - EKS job trigger endpoint * - EKS_TRANSFORM_JOB_NAME - Transformation job name * - BEDROCK_EVALATION_DATA_TRANSFORMATION_ECR_IMAGE_REPO_URI - Container image for evaluation * - CHATTEROPS_BASE_URL - ChatterOps integration URL * - BEDROCK_EVALUATION_DATA_TRANSFORMATION_S3_BUCKET_NAME - S3 bucket for evaluation data * - DATA_TRANSFORMATION_EKS_CLUSTER_AWS_REGION_NAME - AWS region for EKS operations * - BEDROCK_EVALUATION_DATA_TRANSFORMATION_JOB_CONTAINER_TYPE - Container type for evaluation jobs * - BEDROCK_EVALUATION_DATA_TRANSFORMATION_JOB_EXECUTION_COMMAND - Job execution command * - BEDROCK_EVALUATION_EKS_JOB_NAMESPACE - Kubernetes namespace for evaluation jobs * - STUDIOOPS_DATABASE_URL - StudioOps database connection **AWS Bedrock Evaluation Execution** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AWS_BEDROCK_ROLE_ARN - IAM role for Bedrock evaluations * - INHOUSE_EVALUATION_ECR_IMAGE_REPO_URI - In-house evaluation service image * - INHOUSE_EVALUATION_EKS_JOB_NAMESPACE - Namespace for in-house evaluation jobs * - INHOUSE_EVALUATION_JOB_NAME - In-house evaluation job name **Resource Requirements**: Default allocation with ClickHouse requiring dedicated resources **Node Placement** - Node Selector: ``node-pool: studiops-node`` - Tolerations: ``workload=studiops:NoSchedule`` **Health Checks** - Web: ``/api/public/health``, ``/api/public/ready`` - Worker: Process-based health monitoring **Dependencies**: PostgreSQL, ClickHouse, Redis, S3, AWS Bedrock, EKS Jobs Managing Service Threads Service ~~~~~~~~~~~~~~~ Conversation thread management and persistence service that handles multi-turn conversations, thread state management, and conversation history storage. **Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - APPLICATION_NAME - Service identifier for logging and monitoring * - APPLICATION_VERSION - Application version for tracking and compatibility * - HOST - Application bind address for container networking * - PORT - Application server port **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DATABASE_URL - PostgreSQL connection string for thread persistence **Service Integration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - FLOW_SERVICE_BASE_URL - Agentic Flow service endpoint for workflow integration **Logging Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - LOG_LEVEL - Application logging verbosity level **Resource Requirements** - CPU: 1000m (requests) / 1290m (limits) - Memory: 1Gi (requests) / 2Gi (limits) **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/api/v1/health`` **Dependencies**: PostgreSQL, Agentic Flow Service Hulaho Service ~~~~~~~~~~~~~~ Business intelligence and analytics dashboard platform built on Apache Superset, providing comprehensive data visualization and reporting capabilities. **Components** - **Superset Node**: Main web application and API server - **Celery Worker**: Background task processing for reports - **Celery Beat**: Scheduled job execution (optional) - **Celery Flower**: Celery monitoring UI (optional) - **WebSocket Server**: Real-time updates support (optional) **Application Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - APP_NAME - Application display name * - WEB_PROXY_URL - Base URL for the application * - FAVICONS - Custom favicon configuration (JSON array) * - APP_ICON - Application logo path * - LOGOUT_REDIRECT_URL - Post-logout redirect destination **Database Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - DB_USER - PostgreSQL username * - DB_PASS - PostgreSQL password * - DB_HOST - PostgreSQL host * - DB_PORT - PostgreSQL port * - DB_NAME - PostgreSQL database name **Redis Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - REDIS_HOST - Redis server hostname for caching and sessions * - REDIS_PORT - Redis server port * - REDIS_PASSWORD - Redis authentication password **Authentication Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - AUTH_TYPE - Authentication method (e.g., AUTH_OAUTH) * - AUTH_USER_REGISTRATION - Allow automatic user registration * - AUTH_USER_REGISTRATION_ROLE - Default role for new users * - OAUTH_USER_PASSWORD - OAuth user default password **Azure AD OAuth Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - OAUTH_PROVIDERS - OAuth provider configuration (JSON object with Azure AD details) **Security Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - TALISMAN_ENABLED - Talisman security headers toggle * - ENABLE_CORS - Cross-Origin Resource Sharing toggle * - HTTP_HEADERS - Custom HTTP headers (JSON object) * - WTF_CSRF_ENABLED - CSRF protection toggle * - SESSION_COOKIE_SAMESITE - Cookie SameSite policy * - SESSION_COOKIE_SECURE - Require HTTPS for cookies * - ENABLE_PROXY_FIX - Proxy header processing toggle **Feature Flags** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - THUMBNAILS - Dashboard thumbnails toggle * - EMBEDDED_SUPERSET - Embedded dashboard capabilities toggle **Thumbnail Configuration** .. list-table:: :header-rows: 1 :widths: 35 65 * - Variable - Description * - THUMBNAIL_CACHE_CONFIG - Redis-based thumbnail caching configuration * - WEBDRIVER_BASEURL - Base URL for screenshot generation * - SCREENSHOT_SELENIUM_HEADSTART - Selenium startup delay in seconds **Resource Requirements** - Superset Node: CPU 1000m, Memory 4096Mi - Celery Worker: CPU 1000m, Memory 4096Mi - Init Job: CPU 512m-1000m, Memory 1024Mi-4096Mi **Node Placement** - Node Selector: ``node-pool: app-node`` - Tolerations: ``workload=app:NoSchedule`` **Health Check**: ``/health`` **Dependencies**: PostgreSQL, Redis, Azure AD OAuth **Initialization** - Database Migration: Automatic database schema upgrades - Admin User Creation: Default admin user with configurable credentials - Role Initialization: Automatic role and permission setup Best Practices -------------- Security ~~~~~~~~ - Use IRSA (IAM Roles for Service Accounts) for AWS service access - Store sensitive data in Kubernetes secrets or HashiCorp Vault - Enable TLS for all external communications - Implement network policies for service isolation Monitoring ~~~~~~~~~~ - Configure health checks for all services - Set up resource limits and requests - Monitor application logs through CloudWatch - Use StudioOps (Langfuse) for LLM observability Scaling ~~~~~~~ - Configure HPA for services with variable load - Use appropriate node selectors and tolerations - Monitor resource utilization and adjust limits - Consider cluster autoscaling for dynamic workloads Maintenance ~~~~~~~~~~~ - Regular backup of PostgreSQL databases - Monitor S3 storage costs and lifecycle policies - Keep container images updated with security patches - Test rollback procedures in staging environment Troubleshooting --------------- Common Issues ~~~~~~~~~~~~~ **Pod Scheduling Issues** .. code-block:: bash kubectl describe pod -n # Check node selectors, tolerations, and resource availability **Service Discovery Problems** .. code-block:: bash kubectl get endpoints -n # Verify service endpoints are populated **Ingress Configuration** .. code-block:: bash kubectl describe ingress -n # Check ALB controller logs and certificate status **Database Connectivity** .. code-block:: bash kubectl exec -it -n -- nc -zv # Test database connectivity from pods Log Analysis ~~~~~~~~~~~~ .. code-block:: bash # Application logs kubectl logs -f deployment/ -n # Previous container logs (for crash analysis) kubectl logs deployment/ -n --previous # Multiple container logs kubectl logs -f deployment/ -c -n