441 lines
18 KiB
Markdown
441 lines
18 KiB
Markdown
|
# Reconciler Architecture Deep Dive
|
||
|
|
|
||
|
|
## 🎯 **Overview**
|
||
|
|
|
||
|
|
Your `freeleaps-devops-reconciler` is a **sophisticated Kubernetes Operator** that orchestrates your entire DevOps infrastructure. It's not just a simple CRD controller - it's a **full-stack DevOps automation platform** that bridges your Git repositories, container registries, Jenkins pipelines, ArgoCD applications, and Kubernetes deployments.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🏗️ **Architecture Overview**
|
||
|
|
|
||
|
|
### **🔄 The Big Picture: How Your Reconciler Works**
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
|
|
│ FRELEAPS DEVOPS RECONCILER ARCHITECTURE │
|
||
|
|
│ │
|
||
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||
|
|
│ │ KUBERNETES OPERATOR (KOPF) │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
|
|
│ │ │ DevOpsProject │ │ ArgoSetting │ │ JenkinsSetting│ │ IngressResource│ │
|
||
|
|
│ │ │ Controller │ │ Controller │ │ Controller │ │ Controller │ │
|
||
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||
|
|
│ │ EXTERNAL SERVICE INTEGRATION │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
|
|
│ │ │ ArgoCD │ │ Jenkins │ │ Docker Hub │ │ GoDaddy │ │
|
||
|
|
│ │ │ Client │ │ Client │ │ Client │ │ Client │ │
|
||
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||
|
|
│ │ MESSAGING & EVENT SYSTEM │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
|
|
│ │ │ RabbitMQ │ │ Heartbeat │ │ Deployment │ │ TTL Monitor │ │
|
||
|
|
│ │ │ Listener │ │ Sender │ │ Monitor │ │ Manager │ │
|
||
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🔧 **Core Components Deep Dive**
|
||
|
|
|
||
|
|
### **1. DevOpsProject Controller** 🏗️
|
||
|
|
|
||
|
|
**What it does:** The **orchestrator-in-chief** that creates your entire DevOps ecosystem.
|
||
|
|
|
||
|
|
**Real Code Example:**
|
||
|
|
```python
|
||
|
|
@kopf.on.create(group=consts.GROUP, version=consts.VERSION, kind=consts.DEVOPS_PROJECT_KIND)
|
||
|
|
def on_devops_proj_created(name: str, namespace: Optional[str], body: Body, logger: Logger, **kwargs):
|
||
|
|
# When you create a DevOpsProject CR, this triggers:
|
||
|
|
# 1. Validates your Git repo and container registry config
|
||
|
|
# 2. Creates ArgoSetting CR (for ArgoCD management)
|
||
|
|
# 3. Creates JenkinsSetting CR (for CI/CD pipelines)
|
||
|
|
# 4. Creates ContainerRegistry CR (for image management)
|
||
|
|
# 5. Creates GitCredentials CR (for authentication)
|
||
|
|
```
|
||
|
|
|
||
|
|
**Your Actual Flow:**
|
||
|
|
```
|
||
|
|
User creates DevOpsProject CR
|
||
|
|
↓
|
||
|
|
Reconciler validates Git repo + container registry
|
||
|
|
↓
|
||
|
|
Creates ArgoSetting CR (manages ArgoCD projects/apps)
|
||
|
|
↓
|
||
|
|
Creates JenkinsSetting CR (manages CI/CD pipelines)
|
||
|
|
↓
|
||
|
|
Creates ContainerRegistry CR (manages Docker images)
|
||
|
|
↓
|
||
|
|
Creates GitCredentials CR (manages authentication)
|
||
|
|
↓
|
||
|
|
Your DevOps ecosystem is ready! 🎉
|
||
|
|
```
|
||
|
|
|
||
|
|
### **2. ArgoSetting Controller** 🚀
|
||
|
|
|
||
|
|
**What it does:** Manages your **ArgoCD infrastructure** - projects, repositories, and applications.
|
||
|
|
|
||
|
|
**Real Code Example:**
|
||
|
|
```python
|
||
|
|
# When ArgoSetting CR is created:
|
||
|
|
for project in as_spec.projects:
|
||
|
|
# Creates ArgoCD Project
|
||
|
|
desired_resources.append(ManagedResource(
|
||
|
|
resource_type="project",
|
||
|
|
resource_id=project.name,
|
||
|
|
description=project.desc,
|
||
|
|
metadata={
|
||
|
|
"source_repos": [repo.url for repo in as_spec.repositories],
|
||
|
|
"destinations": [{"server": dest.server, "namespace": dest.namespace}]
|
||
|
|
}
|
||
|
|
))
|
||
|
|
|
||
|
|
for app in as_spec.applications:
|
||
|
|
# Creates ArgoCD Application
|
||
|
|
desired_resources.append(ManagedResource(
|
||
|
|
resource_type="application",
|
||
|
|
resource_id=app.name,
|
||
|
|
metadata={
|
||
|
|
"project": app.project,
|
||
|
|
"repo_url": app.source.repo_url,
|
||
|
|
"path": app.source.path,
|
||
|
|
"target_revision": app.source.revision
|
||
|
|
}
|
||
|
|
))
|
||
|
|
```
|
||
|
|
|
||
|
|
**Your Actual ArgoCD Management:**
|
||
|
|
```
|
||
|
|
ArgoSetting CR created
|
||
|
|
↓
|
||
|
|
Reconciler connects to ArgoCD API (argo.mathmast.com)
|
||
|
|
↓
|
||
|
|
Creates ArgoCD Project (defines permissions, repos, destinations)
|
||
|
|
↓
|
||
|
|
Creates ArgoCD Repository (connects to your Git repo)
|
||
|
|
↓
|
||
|
|
Creates ArgoCD Application (deploys your app)
|
||
|
|
↓
|
||
|
|
ArgoCD starts syncing your application! 🔄
|
||
|
|
```
|
||
|
|
|
||
|
|
### **3. JenkinsSetting Controller** ⚙️
|
||
|
|
|
||
|
|
**What it does:** Manages your **Jenkins CI/CD pipelines** - creates folders, pipelines, and credentials.
|
||
|
|
|
||
|
|
**Real Code Example:**
|
||
|
|
```python
|
||
|
|
@kopf.timer(group=consts.GROUP, version=consts.VERSION, kind=consts.JENKINS_SETTINGS_KIND, interval=300)
|
||
|
|
def poll_project_config(name: str, namespace: str, body: Body, logger: logging.Logger, **kwargs):
|
||
|
|
# Every 5 minutes, the reconciler:
|
||
|
|
# 1. Fetches your project's YAML config from Git
|
||
|
|
# 2. Generates Jenkins Pipeline DSL
|
||
|
|
# 3. Creates/updates Jenkins pipelines
|
||
|
|
# 4. Manages pipeline credentials
|
||
|
|
```
|
||
|
|
|
||
|
|
**Your Actual Jenkins Management:**
|
||
|
|
```
|
||
|
|
JenkinsSetting CR created
|
||
|
|
↓
|
||
|
|
Reconciler clones your Git repo
|
||
|
|
↓
|
||
|
|
Reads your project's YAML configuration
|
||
|
|
↓
|
||
|
|
Generates Jenkins Pipeline DSL (Groovy script)
|
||
|
|
↓
|
||
|
|
Creates Jenkins folder structure (project/environment)
|
||
|
|
↓
|
||
|
|
Creates Jenkins pipeline with your DSL
|
||
|
|
↓
|
||
|
|
Your CI/CD pipeline is ready! 🚀
|
||
|
|
```
|
||
|
|
|
||
|
|
### **4. DeploymentRecord Controller** 🎯
|
||
|
|
|
||
|
|
**What it does:** Orchestrates **actual deployments** - triggers Jenkins builds, monitors ArgoCD sync, manages TTL.
|
||
|
|
|
||
|
|
**Real Code Example:**
|
||
|
|
```python
|
||
|
|
@kopf.on.create(group=consts.GROUP, version=consts.VERSION, plural=consts.DEPLOYMENT_RECORD_PLURAL)
|
||
|
|
async def create_deployment_record(spec: Dict[str, Any], name: str, namespace: str, uid: str, logger: Logger, **kwargs):
|
||
|
|
# When you trigger a deployment:
|
||
|
|
# 1. Validates deployment request
|
||
|
|
# 2. Triggers Jenkins build
|
||
|
|
# 3. Monitors build progress
|
||
|
|
# 4. Triggers ArgoCD sync
|
||
|
|
# 5. Monitors deployment status
|
||
|
|
# 6. Manages TTL (Time To Live)
|
||
|
|
```
|
||
|
|
|
||
|
|
**Your Actual Deployment Flow:**
|
||
|
|
```
|
||
|
|
User clicks "Deploy" button
|
||
|
|
↓
|
||
|
|
DeploymentRecord CR created
|
||
|
|
↓
|
||
|
|
Reconciler triggers Jenkins build
|
||
|
|
↓
|
||
|
|
Monitors build phases (building, testing, packaging)
|
||
|
|
↓
|
||
|
|
Triggers ArgoCD sync when build completes
|
||
|
|
↓
|
||
|
|
Monitors ArgoCD sync status
|
||
|
|
↓
|
||
|
|
Creates IngressResource for external access
|
||
|
|
↓
|
||
|
|
Deployment is live! 🌐
|
||
|
|
```
|
||
|
|
|
||
|
|
### **5. IngressResource Controller** 🌐
|
||
|
|
|
||
|
|
**What it does:** Manages **external access** - DNS records, SSL certificates, and ingress rules.
|
||
|
|
|
||
|
|
**Real Code Example:**
|
||
|
|
```python
|
||
|
|
async def create_ingress_resource(self, body: Body, name: str, namespace: str, **kwargs):
|
||
|
|
# When IngressResource CR is created:
|
||
|
|
# 1. Creates DNS record via GoDaddy API
|
||
|
|
# 2. Requests SSL certificate via cert-manager
|
||
|
|
# 3. Creates Kubernetes Ingress
|
||
|
|
# 4. Updates deployment URL
|
||
|
|
# 5. Sends heartbeat with live URL
|
||
|
|
```
|
||
|
|
|
||
|
|
**Your Actual Ingress Management:**
|
||
|
|
```
|
||
|
|
IngressResource CR created
|
||
|
|
↓
|
||
|
|
Reconciler calls GoDaddy API
|
||
|
|
↓
|
||
|
|
Creates DNS record (app.mathmast.com → 4.155.160.32)
|
||
|
|
↓
|
||
|
|
Requests SSL certificate from Let's Encrypt
|
||
|
|
↓
|
||
|
|
Creates Kubernetes Ingress rule
|
||
|
|
↓
|
||
|
|
Your app is accessible at https://app.mathmast.com! 🔒
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 📡 **Messaging & Event System**
|
||
|
|
|
||
|
|
### **RabbitMQ Integration** 🐰
|
||
|
|
|
||
|
|
Your reconciler uses **RabbitMQ** for asynchronous communication and event-driven architecture.
|
||
|
|
|
||
|
|
**Event Types:**
|
||
|
|
```python
|
||
|
|
class EventType(Enum):
|
||
|
|
DEVOPS_INITIALIZE = "DevOpsInitialize" # New project setup
|
||
|
|
DEVOPS_RECONCILE = "DevOpsReconcile" # Deployment trigger
|
||
|
|
DEVOPS_RECONCILE_HEARTBEAT = "DevOpsReconcileJobHeartbeat" # Progress updates
|
||
|
|
```
|
||
|
|
|
||
|
|
**Real Event Flow:**
|
||
|
|
```
|
||
|
|
User triggers deployment
|
||
|
|
↓
|
||
|
|
DevOpsReconcileEvent sent to RabbitMQ
|
||
|
|
↓
|
||
|
|
Reconciler picks up event
|
||
|
|
↓
|
||
|
|
Creates DeploymentRecord CR
|
||
|
|
↓
|
||
|
|
Sends heartbeat every 30 seconds
|
||
|
|
↓
|
||
|
|
User sees real-time progress! 📊
|
||
|
|
```
|
||
|
|
|
||
|
|
### **Heartbeat System** 💓
|
||
|
|
|
||
|
|
**What it does:** Provides **real-time deployment status** to your users.
|
||
|
|
|
||
|
|
**Real Code Example:**
|
||
|
|
```python
|
||
|
|
@dataclass
|
||
|
|
class DevOpsReconcileJobHeartbeatEvent:
|
||
|
|
operation: str = "heartbeat"
|
||
|
|
id: str = "" # deployment ID
|
||
|
|
status: str = "running" # running/success/failed/terminated
|
||
|
|
phase: str = "initializing" # current deployment phase
|
||
|
|
phase_message: str = "" # human-readable message
|
||
|
|
url: Optional[str] = None # live URL when deployment completes
|
||
|
|
```
|
||
|
|
|
||
|
|
**Your Actual Heartbeat Flow:**
|
||
|
|
```
|
||
|
|
Deployment starts
|
||
|
|
↓
|
||
|
|
Heartbeat every 30 seconds:
|
||
|
|
- Phase: "initializing" → "building" → "deploying" → "verifying"
|
||
|
|
- Status: "running" → "success"
|
||
|
|
- URL: None → "https://app.mathmast.com"
|
||
|
|
↓
|
||
|
|
User sees live progress in UI! 📈
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🕒 **TTL (Time To Live) Management**
|
||
|
|
|
||
|
|
### **Automatic Cleanup** 🧹
|
||
|
|
|
||
|
|
Your reconciler includes **sophisticated TTL management** for temporary deployments.
|
||
|
|
|
||
|
|
**Real Code Example:**
|
||
|
|
```python
|
||
|
|
@dataclass
|
||
|
|
class TTLMonitoringState:
|
||
|
|
deployment_record_name: str
|
||
|
|
enabled: bool
|
||
|
|
ttl_seconds: int # Default: 3 hours (10800 seconds)
|
||
|
|
start_time: datetime
|
||
|
|
expiration_time: datetime
|
||
|
|
phase: TTLMonitoringPhase # monitoring/cleanup/completed
|
||
|
|
```
|
||
|
|
|
||
|
|
**Your Actual TTL Flow:**
|
||
|
|
```
|
||
|
|
Deployment completes
|
||
|
|
↓
|
||
|
|
TTL monitoring starts (3 hours by default)
|
||
|
|
↓
|
||
|
|
Every minute: Check if TTL expired
|
||
|
|
↓
|
||
|
|
When expired: Trigger cleanup
|
||
|
|
↓
|
||
|
|
Delete ArgoCD applications
|
||
|
|
↓
|
||
|
|
Delete Kubernetes resources
|
||
|
|
↓
|
||
|
|
Delete DNS records
|
||
|
|
↓
|
||
|
|
Delete SSL certificates
|
||
|
|
↓
|
||
|
|
Environment cleaned up! 🧹
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🔗 **Relationship with Your DevOps Infrastructure**
|
||
|
|
|
||
|
|
### **How the Reconciler Extends Kubernetes** 🔧
|
||
|
|
|
||
|
|
Your reconciler doesn't just manage Kubernetes resources - it **extends Kubernetes** with custom DevOps capabilities:
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
|
|
│ KUBERNETES API EXTENSION │
|
||
|
|
│ │
|
||
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||
|
|
│ │ CUSTOM RESOURCES (CRs) │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ DevOpsProject CR → ArgoSetting CR → JenkinsSetting CR │ │
|
||
|
|
│ │ ↓ ↓ ↓ │ │
|
||
|
|
│ │ ContainerRegistry CR → GitCredentials CR → IngressResource CR │ │
|
||
|
|
│ │ ↓ ↓ ↓ │ │
|
||
|
|
│ │ DeploymentRecord CR → TTL Management → Heartbeat System │ │
|
||
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||
|
|
│ │ EXTERNAL SERVICE ORCHESTRATION │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ ArgoCD API → Jenkins API → Docker Hub API → GoDaddy API │ │
|
||
|
|
│ │ ↓ ↓ ↓ ↓ │ │
|
||
|
|
│ │ Git Repos → CI/CD Pipelines → Container Images → DNS Records │ │
|
||
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
### **Your Actual DevOps Workflow** 🔄
|
||
|
|
|
||
|
|
**Complete Flow Example:**
|
||
|
|
```
|
||
|
|
1. Developer pushes code to Git
|
||
|
|
↓
|
||
|
|
2. User creates DevOpsProject CR
|
||
|
|
↓
|
||
|
|
3. Reconciler creates ArgoSetting CR
|
||
|
|
↓
|
||
|
|
4. Reconciler creates JenkinsSetting CR
|
||
|
|
↓
|
||
|
|
5. Jenkins pipeline created and triggered
|
||
|
|
↓
|
||
|
|
6. Build completes, image pushed to registry
|
||
|
|
↓
|
||
|
|
7. ArgoCD syncs new image
|
||
|
|
↓
|
||
|
|
8. IngressResource creates external access
|
||
|
|
↓
|
||
|
|
9. App is live at https://app.mathmast.com
|
||
|
|
↓
|
||
|
|
10. TTL monitoring starts (3 hours)
|
||
|
|
↓
|
||
|
|
11. After 3 hours: Automatic cleanup
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🎯 **Key Benefits of Your Reconciler Architecture**
|
||
|
|
|
||
|
|
### **1. Declarative DevOps** 📝
|
||
|
|
- **Define once, deploy everywhere**: Your DevOpsProject CR defines your entire infrastructure
|
||
|
|
- **GitOps workflow**: Everything is version-controlled and declarative
|
||
|
|
- **Consistency**: Same process for alpha and production environments
|
||
|
|
|
||
|
|
### **2. Automation at Scale** 🤖
|
||
|
|
- **Zero manual intervention**: From Git push to live deployment
|
||
|
|
- **Multi-environment support**: Alpha and production with same configuration
|
||
|
|
- **Automatic cleanup**: TTL management prevents resource waste
|
||
|
|
|
||
|
|
### **3. Real-time Visibility** 👁️
|
||
|
|
- **Live progress tracking**: Heartbeat system shows real-time deployment status
|
||
|
|
- **Comprehensive monitoring**: Every phase is tracked and reported
|
||
|
|
- **Error handling**: Detailed error messages and recovery mechanisms
|
||
|
|
|
||
|
|
### **4. Enterprise Integration** 🏢
|
||
|
|
- **Multi-service orchestration**: ArgoCD, Jenkins, Docker Hub, GoDaddy
|
||
|
|
- **Security**: Credential management and SSL certificate automation
|
||
|
|
- **Scalability**: Kubernetes-native architecture scales with your cluster
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🔍 **Your Reconciler vs. Traditional DevOps**
|
||
|
|
|
||
|
|
### **Traditional DevOps** 🏗️
|
||
|
|
```
|
||
|
|
Manual Jenkins setup → Manual ArgoCD config → Manual DNS setup → Manual SSL setup
|
||
|
|
```
|
||
|
|
|
||
|
|
### **Your Reconciler** 🚀
|
||
|
|
```
|
||
|
|
DevOpsProject CR → Automatic Jenkins + ArgoCD + DNS + SSL setup
|
||
|
|
```
|
||
|
|
|
||
|
|
**The difference:** Your reconciler transforms **manual DevOps tasks** into **declarative, automated, and scalable** operations that run on Kubernetes.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🎉 **Conclusion**
|
||
|
|
|
||
|
|
Your `freeleaps-devops-reconciler` is not just a Kubernetes operator - it's a **complete DevOps automation platform** that:
|
||
|
|
|
||
|
|
1. **Extends Kubernetes** with custom DevOps capabilities
|
||
|
|
2. **Orchestrates multiple external services** (ArgoCD, Jenkins, Docker Hub, GoDaddy)
|
||
|
|
3. **Provides real-time visibility** into deployment progress
|
||
|
|
4. **Automates complex workflows** from Git push to live deployment
|
||
|
|
5. **Manages the complete lifecycle** including cleanup and TTL
|
||
|
|
|
||
|
|
It's the **brain** of your DevOps infrastructure, making complex multi-service orchestration as simple as creating a Kubernetes Custom Resource! 🧠✨
|