Terraform Installation

This guide covers deploying OneByZero Neo on enterprise AWS accounts using Terraform and Helm for infrastructure provisioning and application deployment.

Overview

OneByZero Neo is a modular AI platform supporting Chatbot, Voice Bot, and Agentic Flow agents. The system runs in a multi-zone Amazon EKS cluster for high availability and uses GitOps (ArgoCD) for continuous deployment.

The platform integrates with:

• Amazon RDS PostgreSQL - Primary relational database
• Amazon OpenSearch - Vector storage and semantic search
• ClickHouse - Time-series analytics database
• Amazon S3 - Object storage for documents and models

Infrastructure as a Service Deployment

OneByZero Neo follows an Infrastructure as a Service (IaaS) deployment model on AWS, where the complete platform is deployed within the enterprise’s own AWS account. This approach provides:

Data Sovereignty : All data remains within the enterprise’s AWS account and VPC. No data is transmitted to external systems, ensuring compliance with data residency requirements.

Network Isolation : The platform operates within private subnets with configurable security groups and network ACLs. External access is controlled through AWS Application Load Balancer with SSL termination.

IAM Integration : Services authenticate to AWS resources using IAM Roles for Service Accounts (IRSA), eliminating the need for long-lived credentials.

Resource Control : Enterprises maintain full control over compute resources, scaling policies, and cost management through AWS native tools.

Deployment Architecture

OneByZero Neo deployment architecture showing the Enterprise AWS Account with VPC, EKS cluster node pools, and AWS managed services.

Prerequisites

Infrastructure Requirements

• Amazon EKS cluster (v1.24+)

• Node groups with appropriate taints and labels:

  • app-node pool for application workloads
  • studiops-node pool for Langfuse/ClickHouse

• AWS Application Load Balancer Controller

• External DNS Controller

• Amazon RDS PostgreSQL instance

• Amazon OpenSearch cluster

• AWS S3 buckets for data storage

• AWS SQS queues for message processing

Required AWS Resources

RDS Database : PostgreSQL instance for application data persistence.

OpenSearch Domain : Managed OpenSearch cluster for vector storage and semantic search.

S3 Buckets

• Data ingestion bucket
• StudioOps bucket
• StudioOps datasets bucket
• Bedrock knowledge base data ingestion bucket
• Supplemental storage bucket
• Voice schedule bucket
• Bedrock evaluators bucket

SQS Queues

• KPI events queue
• Voice calls queue

Security Requirements

• IAM Role for EKS IRSA (IAM Roles for Service Accounts)
• SSL Certificate from AWS Certificate Manager (ACM)
• Application Load Balancer for ingress traffic

Password Generation Guidelines

Database passwords (RDS PostgreSQL, Redis, ClickHouse) should use URL-safe characters to prevent connection string parsing issues.

Safe Characters (No Encoding Required)

A-Z  a-z  0-9  _  -  .  ~

Characters to Avoid

@  :  /  ?  #  &  =  %  +  (space)  ;  "  '  <  >  \  ^  `  {  }  [  ]

If special characters must be used, URL-encode the password before storing:

python3 -c "import urllib.parse; print(urllib.parse.quote_plus('your-password'))"

Keys Generation

Generate required encryption keys and secrets before deployment:

JWT Key Pair Generation

# Generate private key
openssl genpkey -algorithm RSA -out private_key.pem

# Generate public key from private key
openssl rsa -pubout -in private_key.pem -out public_key.pem

Helm Deployment Process

Pre-deployment Validation

# Verify cluster connectivity
kubectl cluster-info

# Check node readiness
kubectl get nodes

# Verify required namespaces
kubectl create namespace <NAMESPACE> --dry-run=client -o yaml | kubectl apply -f -

# Validate AWS Load Balancer Controller
kubectl get deployment -n kube-system aws-load-balancer-controller

Deploy Infrastructure Services

Deploy infrastructure services first as they are dependencies for application services.

Voice Redis

helm upgrade --install voice-redis ./voice_redis \
  --namespace <NAMESPACE> \
  --create-namespace \
  --wait \
  --timeout 10m

Langfuse (StudioOps)

helm upgrade --install langfuse ./langfuse \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 15m

Hulaho (Analytics Dashboard)

helm upgrade --install hulaho ./hulaho \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 15m

Deploy Core Application Services

Deploy services in dependency order:

# EKS Jobs Managing Service (required by Knowledge Base)
helm upgrade --install eks-jobs-managing ./eks_jobs_managing_service \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

# Knowledge Base Service
helm upgrade --install knowledge-base ./knowledge_base \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

# Agentic Flow Service
helm upgrade --install agentic-flow ./agentic_flow \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

# Bot Builder Services
helm upgrade --install botbuilder ./botbuilder \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

# KPI Events Polling Service
helm upgrade --install kpi-events-polling ./kpi_events_polling_service \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

# Threads Service
helm upgrade --install threads-service ./threads_service \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

Deploy Voice Services

# Voice Backend
helm upgrade --install voice-backend ./voice_backend \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

# Voice Agent
helm upgrade --install voice-agent ./voice_agent \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

# Voice Scheduler
helm upgrade --install voice-scheduler ./voice_scheduler \
  --namespace <NAMESPACE> \
  --wait \
  --timeout 10m

Verification

# Check all deployments
kubectl get deployments -n <NAMESPACE>

# Check all services
kubectl get services -n <NAMESPACE>

# Check ingress resources
kubectl get ingress -n <NAMESPACE>

# Check pod status
kubectl get pods -n <NAMESPACE>

# View logs for troubleshooting
kubectl logs -f deployment/<service-name> -n <NAMESPACE>

Upgrade and Rollback

Upgrade Process

helm upgrade <release-name> ./<chart-directory> \
  --namespace <NAMESPACE> \
  --reuse-values \
  --wait \
  --timeout 10m

Rollback Process

# List release history
helm history <release-name> --namespace <NAMESPACE>

# Rollback to previous version
helm rollback <release-name> --namespace <NAMESPACE>

# Rollback to specific revision
helm rollback <release-name> <revision-number> --namespace <NAMESPACE>

Service Configuration Reference

This section documents all required environment variables for each microservice. Variables not listed are considered optional and use default values.

Note

All secrets should be stored in HashiCorp Vault or Kubernetes Secrets and mapped to deployments using the VaultStaticSecrets CRD or native Kubernetes secret references.

Agentic Flow Service

AI workflow orchestration and LangFlow integration service that manages AI workflows and provides API endpoints for conversational AI applications.

Network & Application Configuration

Database Configuration

Logging & Monitoring

API Integration

Feature Flags

OpenSearch Configuration

AWS Bedrock Knowledge Base Configuration

Resource Requirements

• CPU: 1000m (requests) / 1290m (limits)
• Memory: 1Gi (requests) / 2Gi (limits)

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /health_check

Dependencies: PostgreSQL, OpenSearch, AWS Bedrock, StudioOps telemetry service

Bot Builder Service

Conversational bot creation and management platform with separate builder and viewer components for designing and running chatbots.

Builder Component - Application Configuration

Authentication & Security

Database Configuration

Bot Behavior Configuration

External Service Integration

Organization & User Management

Session Management

Metrics & Analytics

AWS S3 Configuration

Service Integration URLs

Analytics Dashboard Integration

Feature Flags

Webhook Configuration

Branding

Viewer Component - Additional Configuration

Resource Requirements (Viewer)

• CPU: 1200m (requests) / 1290m (limits)
• Memory: 1Gi (requests) / 2Gi (limits)

Autoscaling (Viewer)

• Min Replicas: 1
• Max Replicas: 2
• Target CPU Utilization: 80%

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Dependencies: PostgreSQL, S3, Azure AD, Voice services, Agentic Flow, Knowledge Base

Channel Event Gateway Service

Receives incoming webhook requests from channels and passes them to the right agent.

Google Chat

Knowledge Base Service

Document ingestion, processing, and retrieval service that manages knowledge bases, handles document uploads, and provides semantic search capabilities.

Application Configuration

Database Configuration

Service Integration

AWS Configuration

S3 Storage Configuration

OpenSearch Configuration

Embedding Model Configuration

Data Ingestion Service Configuration

AI Model Configuration

AWS Bedrock Configuration

Bedrock Knowledge Base Configuration

Bedrock Knowledge Base OpenSearch Configuration

Resource Requirements

• CPU: 1200m (requests) / 1290m (limits)
• Memory: 1Gi (requests) / 2Gi (limits)

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /api/v1/health

Dependencies: OpenSearch, S3, AWS Bedrock, EKS Jobs Managing Service, PostgreSQL

Voice Agent Service

Real-time voice conversation handling service that manages voice calls, integrates with speech services, and provides conversational AI capabilities.

Agent Configuration

Database Configuration

AWS Integration

Azure OpenAI Configuration

Azure Speech Services Configuration

ElevenLabs Voice Synthesis Configuration

Twilio Telephony Configuration

Redis Configuration

Conversation Configuration

Caching Configuration

Logging Configuration

Telemetry Configuration

Conversation Schema Configuration

Resource Requirements

• CPU: 250m (requests) / 500m (limits)
• Memory: 512Mi (requests) / 3Gi (limits)

Autoscaling

• Min Replicas: 1
• Max Replicas: 1
• Target CPU Utilization: 70%
• Target Memory Utilization: 80%

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /

Dependencies: Redis, PostgreSQL, Azure OpenAI, Azure Speech, ElevenLabs, Twilio, SQS

Voice Backend Service

Voice call management and scheduling backend service that handles voice call orchestration, scheduling, and integration with telephony services.

AWS Integration

Database Configuration

ElevenLabs Voice Configuration

Twilio Configuration

Azure OpenAI Configuration

Analytics Integration

Scheduling Configuration

Service Configuration

Resource Requirements

• CPU: 1000m (requests) / 1290m (limits)
• Memory: 1Gi (requests) / 2Gi (limits)

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /health

Dependencies: SQS, S3, ElevenLabs, Twilio, PostgreSQL, Azure OpenAI

Voice Scheduler Service

Automated voice call scheduling and management service that handles call timing, contact policies, and queue management.

Application Configuration

Scheduling Configuration

Contact Policy Configuration

Database Configuration

Azure Service Bus Configuration

AWS SQS Configuration

Service Integration

Logging Configuration

Resource Requirements

• CPU: 1000m (requests) / 1290m (limits)
• Memory: 1Gi (requests) / 2Gi (limits)

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /health

Dependencies: Azure Service Bus, SQS, PostgreSQL, Voice Agent Service

KPI Events Polling Service

Analytics and metrics collection service that processes platform events and generates insights using AI models.

Application Configuration

Database Configuration

Queue Configuration

AWS Bedrock Model Configuration

Telemetry Configuration

Logging Configuration

Resource Requirements

• CPU: 200m (requests) / 1290m (limits)
• Memory: 512Mi (requests) / 2Gi (limits)

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /api/v1/health

Dependencies: SQS, AWS Bedrock, PostgreSQL, StudioOps

EKS Jobs Managing Service

Kubernetes job orchestration and management service that handles batch processing and job lifecycle management.

Application Configuration

Resource Requirements: Default allocation

Node Placement: Default (no specific node selector configured)

Health Check: /api/v1/health

Dependencies: Kubernetes API, RBAC permissions for job management

Voice Redis Service

In-memory data store for voice services session management and caching.

Redis Configuration

Resource Requirements

• CPU: 1200m (requests) / 1290m (limits)
• Memory: 1Gi (requests) / 2Gi (limits)

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: Redis ping command

Dependencies: None (standalone service)

Langfuse (StudioOps) Service

LLM observability, analytics, and evaluation platform providing comprehensive monitoring and analysis of AI model performance.

Components

• Web Interface: Dashboard and API endpoints
• Worker: Background processing for analytics
• ClickHouse: Time-series database for metrics storage
• PostgreSQL: Metadata and configuration storage

Core Configuration

Database Configuration

ClickHouse Configuration

Feature Flags

AWS S3 Dataset Configuration

Bedrock Evaluation Configuration

AWS Bedrock Evaluation Execution

Resource Requirements: Default allocation with ClickHouse requiring dedicated resources

Node Placement

• Node Selector: node-pool: studiops-node
• Tolerations: workload=studiops:NoSchedule

Health Checks

• Web: /api/public/health, /api/public/ready
• Worker: Process-based health monitoring

Dependencies: PostgreSQL, ClickHouse, Redis, S3, AWS Bedrock, EKS Jobs Managing Service

Threads Service

Conversation thread management and persistence service that handles multi-turn conversations, thread state management, and conversation history storage.

Application Configuration

Database Configuration

Service Integration

Logging Configuration

Resource Requirements

• CPU: 1000m (requests) / 1290m (limits)
• Memory: 1Gi (requests) / 2Gi (limits)

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /api/v1/health

Dependencies: PostgreSQL, Agentic Flow Service

Hulaho Service

Business intelligence and analytics dashboard platform built on Apache Superset, providing comprehensive data visualization and reporting capabilities.

Components

• Superset Node: Main web application and API server
• Celery Worker: Background task processing for reports
• Celery Beat: Scheduled job execution (optional)
• Celery Flower: Celery monitoring UI (optional)
• WebSocket Server: Real-time updates support (optional)

Application Configuration

Database Configuration

Redis Configuration

Authentication Configuration

Azure AD OAuth Configuration

Security Configuration

Feature Flags

Thumbnail Configuration

Resource Requirements

• Superset Node: CPU 1000m, Memory 4096Mi
• Celery Worker: CPU 1000m, Memory 4096Mi
• Init Job: CPU 512m-1000m, Memory 1024Mi-4096Mi

Node Placement

• Node Selector: node-pool: app-node
• Tolerations: workload=app:NoSchedule

Health Check: /health

Dependencies: PostgreSQL, Redis, Azure AD OAuth

Initialization

• Database Migration: Automatic database schema upgrades
• Admin User Creation: Default admin user with configurable credentials
• Role Initialization: Automatic role and permission setup

Best Practices

Security

• Use IRSA (IAM Roles for Service Accounts) for AWS service access
• Store sensitive data in Kubernetes secrets or HashiCorp Vault
• Enable TLS for all external communications
• Implement network policies for service isolation

Monitoring

• Configure health checks for all services
• Set up resource limits and requests
• Monitor application logs through CloudWatch
• Use StudioOps (Langfuse) for LLM observability

Scaling

• Configure HPA for services with variable load
• Use appropriate node selectors and tolerations
• Monitor resource utilization and adjust limits
• Consider cluster autoscaling for dynamic workloads

Maintenance

• Regular backup of PostgreSQL databases
• Monitor S3 storage costs and lifecycle policies
• Keep container images updated with security patches
• Test rollback procedures in staging environment

Troubleshooting

Common Issues

Pod Scheduling Issues

kubectl describe pod <pod-name> -n <namespace>
# Check node selectors, tolerations, and resource availability

Service Discovery Problems

kubectl get endpoints -n <namespace>
# Verify service endpoints are populated

Ingress Configuration

kubectl describe ingress -n <namespace>
# Check ALB controller logs and certificate status

Database Connectivity

kubectl exec -it <pod-name> -n <namespace> -- nc -zv <db-host> <db-port>
# Test database connectivity from pods

Log Analysis

# Application logs
kubectl logs -f deployment/<service-name> -n <namespace>

# Previous container logs (for crash analysis)
kubectl logs deployment/<service-name> -n <namespace> --previous

# Multiple container logs
kubectl logs -f deployment/<service-name> -c <container-name> -n <namespace>