SEO_iamge_renamer_starting_point/packages/worker/README.md

# SEO Image Renamer Worker Service

A production-ready NestJS worker service that processes images using AI vision analysis to generate SEO-optimized filenames.

## Features

### 🤖 AI Vision Analysis
- **OpenAI GPT-4 Vision**: Advanced image understanding with custom prompts
- **Google Cloud Vision**: Label detection with confidence scoring
- **Fallback Strategy**: Automatic failover between providers
- **Rate Limiting**: Respects API quotas with intelligent throttling

### 🖼️ Image Processing Pipeline
- **File Validation**: Format validation and virus scanning
- **Metadata Extraction**: EXIF, IPTC, and XMP data preservation
- **Image Optimization**: Sharp-powered processing with quality control
- **Format Support**: JPG, PNG, GIF, WebP with conversion capabilities

### 📦 Storage Integration
- **MinIO Support**: S3-compatible object storage
- **AWS S3 Support**: Native AWS integration
- **Temporary Files**: Automatic cleanup and management
- **ZIP Creation**: Batch downloads with EXIF preservation

### 🔒 Security Features
- **Virus Scanning**: ClamAV integration for file safety
- **File Validation**: Comprehensive format and size checking
- **Quarantine System**: Automatic threat isolation
- **Security Logging**: Incident tracking and alerting

### ⚡ Queue Processing
- **BullMQ Integration**: Reliable job processing with Redis
- **Retry Logic**: Exponential backoff with intelligent failure handling
- **Progress Tracking**: Real-time WebSocket updates
- **Batch Processing**: Efficient multi-image workflows

### 📊 Monitoring & Observability
- **Prometheus Metrics**: Comprehensive performance monitoring
- **Health Checks**: Kubernetes-ready health endpoints
- **Structured Logging**: Winston-powered logging with rotation
- **Error Tracking**: Detailed error reporting and analysis

## Quick Start

### Development Setup

1. **Clone and Install**
   ```bash
   cd packages/worker
   npm install
   ```

2. **Environment Configuration**
   ```bash
   cp .env.example .env
   # Edit .env with your configuration
   ```

3. **Start Dependencies**
   ```bash
   docker-compose up redis minio -d
   ```

4. **Run Development Server**
   ```bash
   npm run start:dev
   ```

### Production Deployment

1. **Docker Compose**
   ```bash
   docker-compose up -d
   ```

2. **Kubernetes**
   ```bash
   kubectl apply -f ../k8s/worker-deployment.yaml
   ```

## Configuration

### Required Environment Variables

```env
# Database
DATABASE_URL=postgresql://user:pass@host:5432/db

# Redis
REDIS_URL=redis://localhost:6379

# AI Vision (at least one required)
OPENAI_API_KEY=your_key
# OR
GOOGLE_CLOUD_VISION_KEY=path/to/service-account.json

# Storage (choose one)
MINIO_ENDPOINT=localhost
MINIO_ACCESS_KEY=access_key
MINIO_SECRET_KEY=secret_key
# OR
AWS_ACCESS_KEY_ID=your_key
AWS_SECRET_ACCESS_KEY=your_secret
AWS_BUCKET_NAME=your_bucket
```

### Optional Configuration

```env
# Processing
MAX_CONCURRENT_JOBS=5
VISION_CONFIDENCE_THRESHOLD=0.40
MAX_FILE_SIZE=52428800

# Security
VIRUS_SCAN_ENABLED=true
CLAMAV_HOST=localhost

# Monitoring
METRICS_ENABLED=true
LOG_LEVEL=info
```

## API Endpoints

### Health Checks
- `GET /health` - Basic health check
- `GET /health/detailed` - Comprehensive system status
- `GET /health/ready` - Kubernetes readiness probe
- `GET /health/live` - Kubernetes liveness probe

### Metrics
- `GET /metrics` - Prometheus metrics endpoint

## Architecture

### Processing Pipeline

```
Image Upload → Virus Scan → Metadata Extraction → AI Analysis → Filename Generation → Database Update
     ↓              ↓              ↓                ↓               ↓                ↓
  Security      Validation    EXIF/IPTC      Vision APIs    SEO Optimization   Progress Update
```

### Queue Structure

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│ image-processing│    │ batch-processing │    │ virus-scan      │
│ - Individual    │    │ - Batch coord.   │    │ - Security      │
│ - AI analysis   │    │ - ZIP creation   │    │ - Quarantine    │
│ - Filename gen. │    │ - Progress agg.  │    │ - Cleanup       │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

## Performance

### Throughput
- **Images/minute**: 50-100 (depending on AI provider limits)
- **Concurrent jobs**: Configurable (default: 5)
- **File size limit**: 50MB (configurable)

### Resource Usage
- **Memory**: ~200MB base + ~50MB per concurrent job
- **CPU**: ~100% per active image processing job
- **Storage**: Temporary files cleaned automatically

## Monitoring

### Key Metrics
- `seo_worker_jobs_total` - Total jobs processed
- `seo_worker_job_duration_seconds` - Processing time distribution
- `seo_worker_vision_api_calls_total` - AI API usage
- `seo_worker_processing_errors_total` - Error rates

### Alerts
- High error rates (>5%)
- API rate limit approaching
- Queue backlog growing
- Storage space low
- Memory usage high

## Troubleshooting

### Common Issues

1. **AI Vision API Failures**
   ```bash
   # Check API keys and quotas
   curl -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/models
   ```

2. **Storage Connection Issues**
   ```bash
   # Test MinIO connection
   mc alias set local http://localhost:9000 access_key secret_key
   mc ls local
   ```

3. **Queue Processing Stopped**
   ```bash
   # Check Redis connection
   redis-cli ping
   
   # Check queue status
   curl http://localhost:3002/health/detailed
   ```

4. **High Memory Usage**
   ```bash
   # Check temp file cleanup
   ls -la /tmp/seo-worker/
   
   # Force cleanup
   curl -X POST http://localhost:3002/admin/cleanup
   ```

### Debugging

Enable debug logging:
```env
LOG_LEVEL=debug
NODE_ENV=development
```

Monitor processing in real-time:
```bash
# Follow logs
docker logs -f seo-worker

# Monitor metrics
curl http://localhost:9090/metrics | grep seo_worker
```

## Development

### Project Structure
```
src/
├── config/          # Configuration and validation
├── vision/          # AI vision services
├── processors/      # BullMQ job processors
├── storage/         # File and cloud storage
├── queue/           # Queue management and tracking
├── security/        # Virus scanning and validation
├── database/        # Database integration
├── monitoring/      # Metrics and logging
└── health/          # Health check endpoints
```

### Testing
```bash
# Unit tests
npm test

# Integration tests
npm run test:e2e

# Coverage report
npm run test:cov
```

### Contributing

1. Fork the repository
2. Create a feature branch
3. Add comprehensive tests
4. Update documentation
5. Submit a pull request

## License

Proprietary - SEO Image Renamer Platform

## Support

For technical support and questions:
- Documentation: [Internal Wiki]
- Issues: [Project Board]
- Contact: engineering@seo-image-renamer.com