SEO_iamge_renamer_starting_point/packages/worker/README.md

SEO Image Renamer Worker Service

A production-ready NestJS worker service that processes images using AI vision analysis to generate SEO-optimized filenames.

Features

🤖 AI Vision Analysis

• OpenAI GPT-4 Vision: Advanced image understanding with custom prompts
• Google Cloud Vision: Label detection with confidence scoring
• Fallback Strategy: Automatic failover between providers
• Rate Limiting: Respects API quotas with intelligent throttling

🖼️ Image Processing Pipeline

• File Validation: Format validation and virus scanning
• Metadata Extraction: EXIF, IPTC, and XMP data preservation
• Image Optimization: Sharp-powered processing with quality control
• Format Support: JPG, PNG, GIF, WebP with conversion capabilities

📦 Storage Integration

• MinIO Support: S3-compatible object storage
• AWS S3 Support: Native AWS integration
• Temporary Files: Automatic cleanup and management
• ZIP Creation: Batch downloads with EXIF preservation

🔒 Security Features

• Virus Scanning: ClamAV integration for file safety
• File Validation: Comprehensive format and size checking
• Quarantine System: Automatic threat isolation
• Security Logging: Incident tracking and alerting

⚡ Queue Processing

• BullMQ Integration: Reliable job processing with Redis
• Retry Logic: Exponential backoff with intelligent failure handling
• Progress Tracking: Real-time WebSocket updates
• Batch Processing: Efficient multi-image workflows

📊 Monitoring & Observability

• Prometheus Metrics: Comprehensive performance monitoring
• Health Checks: Kubernetes-ready health endpoints
• Structured Logging: Winston-powered logging with rotation
• Error Tracking: Detailed error reporting and analysis

Quick Start

Development Setup

1. Clone and Install

   cd packages/worker
   npm install
   

2. Environment Configuration

   cp .env.example .env
   # Edit .env with your configuration
   

3. Start Dependencies

   docker-compose up redis minio -d
   

4. Run Development Server

   npm run start:dev
   

Production Deployment

1. Docker Compose

   docker-compose up -d
   

2. Kubernetes

   kubectl apply -f ../k8s/worker-deployment.yaml
   

Configuration

Required Environment Variables

# Database
DATABASE_URL=postgresql://user:pass@host:5432/db

# Redis
REDIS_URL=redis://localhost:6379

# AI Vision (at least one required)
OPENAI_API_KEY=your_key
# OR
GOOGLE_CLOUD_VISION_KEY=path/to/service-account.json

# Storage (choose one)
MINIO_ENDPOINT=localhost
MINIO_ACCESS_KEY=access_key
MINIO_SECRET_KEY=secret_key
# OR
AWS_ACCESS_KEY_ID=your_key
AWS_SECRET_ACCESS_KEY=your_secret
AWS_BUCKET_NAME=your_bucket

Optional Configuration

# Processing
MAX_CONCURRENT_JOBS=5
VISION_CONFIDENCE_THRESHOLD=0.40
MAX_FILE_SIZE=52428800

# Security
VIRUS_SCAN_ENABLED=true
CLAMAV_HOST=localhost

# Monitoring
METRICS_ENABLED=true
LOG_LEVEL=info

API Endpoints

Health Checks

• GET /health - Basic health check
• GET /health/detailed - Comprehensive system status
• GET /health/ready - Kubernetes readiness probe
• GET /health/live - Kubernetes liveness probe

Metrics

• GET /metrics - Prometheus metrics endpoint

Architecture

Processing Pipeline

Image Upload → Virus Scan → Metadata Extraction → AI Analysis → Filename Generation → Database Update
     ↓              ↓              ↓                ↓               ↓                ↓
  Security      Validation    EXIF/IPTC      Vision APIs    SEO Optimization   Progress Update

Queue Structure

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│ image-processing│    │ batch-processing │    │ virus-scan      │
│ - Individual    │    │ - Batch coord.   │    │ - Security      │
│ - AI analysis   │    │ - ZIP creation   │    │ - Quarantine    │
│ - Filename gen. │    │ - Progress agg.  │    │ - Cleanup       │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Performance

Throughput

• Images/minute: 50-100 (depending on AI provider limits)
• Concurrent jobs: Configurable (default: 5)
• File size limit: 50MB (configurable)

Resource Usage

• Memory: ~200MB base + ~50MB per concurrent job
• CPU: ~100% per active image processing job
• Storage: Temporary files cleaned automatically

Monitoring

Key Metrics

• seo_worker_jobs_total - Total jobs processed
• seo_worker_job_duration_seconds - Processing time distribution
• seo_worker_vision_api_calls_total - AI API usage
• seo_worker_processing_errors_total - Error rates

Alerts

• High error rates (>5%)
• API rate limit approaching
• Queue backlog growing
• Storage space low
• Memory usage high

Troubleshooting

Common Issues

1. AI Vision API Failures

   # Check API keys and quotas
   curl -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/models
   

2. Storage Connection Issues

   # Test MinIO connection
   mc alias set local http://localhost:9000 access_key secret_key
   mc ls local
   

3. Queue Processing Stopped

   # Check Redis connection
   redis-cli ping
   
   # Check queue status
   curl http://localhost:3002/health/detailed
   

4. High Memory Usage

   # Check temp file cleanup
   ls -la /tmp/seo-worker/
   
   # Force cleanup
   curl -X POST http://localhost:3002/admin/cleanup
   

Debugging

Enable debug logging:

LOG_LEVEL=debug
NODE_ENV=development

Monitor processing in real-time:

# Follow logs
docker logs -f seo-worker

# Monitor metrics
curl http://localhost:9090/metrics | grep seo_worker

Development

Project Structure

src/
├── config/          # Configuration and validation
├── vision/          # AI vision services
├── processors/      # BullMQ job processors
├── storage/         # File and cloud storage
├── queue/           # Queue management and tracking
├── security/        # Virus scanning and validation
├── database/        # Database integration
├── monitoring/      # Metrics and logging
└── health/          # Health check endpoints

Testing

# Unit tests
npm test

# Integration tests
npm run test:e2e

# Coverage report
npm run test:cov

Contributing

1. Fork the repository
2. Create a feature branch
3. Add comprehensive tests
4. Update documentation
5. Submit a pull request

License

Proprietary - SEO Image Renamer Platform

Support

For technical support and questions:

• Documentation: [Internal Wiki]
• Issues: [Project Board]
• Contact: engineering@seo-image-renamer.com