280 lines
7 KiB
Markdown
280 lines
7 KiB
Markdown
![]() |
# SEO Image Renamer Worker Service
|
||
|
|
||
|
A production-ready NestJS worker service that processes images using AI vision analysis to generate SEO-optimized filenames.
|
||
|
|
||
|
## Features
|
||
|
|
||
|
### 🤖 AI Vision Analysis
|
||
|
- **OpenAI GPT-4 Vision**: Advanced image understanding with custom prompts
|
||
|
- **Google Cloud Vision**: Label detection with confidence scoring
|
||
|
- **Fallback Strategy**: Automatic failover between providers
|
||
|
- **Rate Limiting**: Respects API quotas with intelligent throttling
|
||
|
|
||
|
### 🖼️ Image Processing Pipeline
|
||
|
- **File Validation**: Format validation and virus scanning
|
||
|
- **Metadata Extraction**: EXIF, IPTC, and XMP data preservation
|
||
|
- **Image Optimization**: Sharp-powered processing with quality control
|
||
|
- **Format Support**: JPG, PNG, GIF, WebP with conversion capabilities
|
||
|
|
||
|
### 📦 Storage Integration
|
||
|
- **MinIO Support**: S3-compatible object storage
|
||
|
- **AWS S3 Support**: Native AWS integration
|
||
|
- **Temporary Files**: Automatic cleanup and management
|
||
|
- **ZIP Creation**: Batch downloads with EXIF preservation
|
||
|
|
||
|
### 🔒 Security Features
|
||
|
- **Virus Scanning**: ClamAV integration for file safety
|
||
|
- **File Validation**: Comprehensive format and size checking
|
||
|
- **Quarantine System**: Automatic threat isolation
|
||
|
- **Security Logging**: Incident tracking and alerting
|
||
|
|
||
|
### ⚡ Queue Processing
|
||
|
- **BullMQ Integration**: Reliable job processing with Redis
|
||
|
- **Retry Logic**: Exponential backoff with intelligent failure handling
|
||
|
- **Progress Tracking**: Real-time WebSocket updates
|
||
|
- **Batch Processing**: Efficient multi-image workflows
|
||
|
|
||
|
### 📊 Monitoring & Observability
|
||
|
- **Prometheus Metrics**: Comprehensive performance monitoring
|
||
|
- **Health Checks**: Kubernetes-ready health endpoints
|
||
|
- **Structured Logging**: Winston-powered logging with rotation
|
||
|
- **Error Tracking**: Detailed error reporting and analysis
|
||
|
|
||
|
## Quick Start
|
||
|
|
||
|
### Development Setup
|
||
|
|
||
|
1. **Clone and Install**
|
||
|
```bash
|
||
|
cd packages/worker
|
||
|
npm install
|
||
|
```
|
||
|
|
||
|
2. **Environment Configuration**
|
||
|
```bash
|
||
|
cp .env.example .env
|
||
|
# Edit .env with your configuration
|
||
|
```
|
||
|
|
||
|
3. **Start Dependencies**
|
||
|
```bash
|
||
|
docker-compose up redis minio -d
|
||
|
```
|
||
|
|
||
|
4. **Run Development Server**
|
||
|
```bash
|
||
|
npm run start:dev
|
||
|
```
|
||
|
|
||
|
### Production Deployment
|
||
|
|
||
|
1. **Docker Compose**
|
||
|
```bash
|
||
|
docker-compose up -d
|
||
|
```
|
||
|
|
||
|
2. **Kubernetes**
|
||
|
```bash
|
||
|
kubectl apply -f ../k8s/worker-deployment.yaml
|
||
|
```
|
||
|
|
||
|
## Configuration
|
||
|
|
||
|
### Required Environment Variables
|
||
|
|
||
|
```env
|
||
|
# Database
|
||
|
DATABASE_URL=postgresql://user:pass@host:5432/db
|
||
|
|
||
|
# Redis
|
||
|
REDIS_URL=redis://localhost:6379
|
||
|
|
||
|
# AI Vision (at least one required)
|
||
|
OPENAI_API_KEY=your_key
|
||
|
# OR
|
||
|
GOOGLE_CLOUD_VISION_KEY=path/to/service-account.json
|
||
|
|
||
|
# Storage (choose one)
|
||
|
MINIO_ENDPOINT=localhost
|
||
|
MINIO_ACCESS_KEY=access_key
|
||
|
MINIO_SECRET_KEY=secret_key
|
||
|
# OR
|
||
|
AWS_ACCESS_KEY_ID=your_key
|
||
|
AWS_SECRET_ACCESS_KEY=your_secret
|
||
|
AWS_BUCKET_NAME=your_bucket
|
||
|
```
|
||
|
|
||
|
### Optional Configuration
|
||
|
|
||
|
```env
|
||
|
# Processing
|
||
|
MAX_CONCURRENT_JOBS=5
|
||
|
VISION_CONFIDENCE_THRESHOLD=0.40
|
||
|
MAX_FILE_SIZE=52428800
|
||
|
|
||
|
# Security
|
||
|
VIRUS_SCAN_ENABLED=true
|
||
|
CLAMAV_HOST=localhost
|
||
|
|
||
|
# Monitoring
|
||
|
METRICS_ENABLED=true
|
||
|
LOG_LEVEL=info
|
||
|
```
|
||
|
|
||
|
## API Endpoints
|
||
|
|
||
|
### Health Checks
|
||
|
- `GET /health` - Basic health check
|
||
|
- `GET /health/detailed` - Comprehensive system status
|
||
|
- `GET /health/ready` - Kubernetes readiness probe
|
||
|
- `GET /health/live` - Kubernetes liveness probe
|
||
|
|
||
|
### Metrics
|
||
|
- `GET /metrics` - Prometheus metrics endpoint
|
||
|
|
||
|
## Architecture
|
||
|
|
||
|
### Processing Pipeline
|
||
|
|
||
|
```
|
||
|
Image Upload → Virus Scan → Metadata Extraction → AI Analysis → Filename Generation → Database Update
|
||
|
↓ ↓ ↓ ↓ ↓ ↓
|
||
|
Security Validation EXIF/IPTC Vision APIs SEO Optimization Progress Update
|
||
|
```
|
||
|
|
||
|
### Queue Structure
|
||
|
|
||
|
```
|
||
|
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||
|
│ image-processing│ │ batch-processing │ │ virus-scan │
|
||
|
│ - Individual │ │ - Batch coord. │ │ - Security │
|
||
|
│ - AI analysis │ │ - ZIP creation │ │ - Quarantine │
|
||
|
│ - Filename gen. │ │ - Progress agg. │ │ - Cleanup │
|
||
|
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
||
|
```
|
||
|
|
||
|
## Performance
|
||
|
|
||
|
### Throughput
|
||
|
- **Images/minute**: 50-100 (depending on AI provider limits)
|
||
|
- **Concurrent jobs**: Configurable (default: 5)
|
||
|
- **File size limit**: 50MB (configurable)
|
||
|
|
||
|
### Resource Usage
|
||
|
- **Memory**: ~200MB base + ~50MB per concurrent job
|
||
|
- **CPU**: ~100% per active image processing job
|
||
|
- **Storage**: Temporary files cleaned automatically
|
||
|
|
||
|
## Monitoring
|
||
|
|
||
|
### Key Metrics
|
||
|
- `seo_worker_jobs_total` - Total jobs processed
|
||
|
- `seo_worker_job_duration_seconds` - Processing time distribution
|
||
|
- `seo_worker_vision_api_calls_total` - AI API usage
|
||
|
- `seo_worker_processing_errors_total` - Error rates
|
||
|
|
||
|
### Alerts
|
||
|
- High error rates (>5%)
|
||
|
- API rate limit approaching
|
||
|
- Queue backlog growing
|
||
|
- Storage space low
|
||
|
- Memory usage high
|
||
|
|
||
|
## Troubleshooting
|
||
|
|
||
|
### Common Issues
|
||
|
|
||
|
1. **AI Vision API Failures**
|
||
|
```bash
|
||
|
# Check API keys and quotas
|
||
|
curl -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/models
|
||
|
```
|
||
|
|
||
|
2. **Storage Connection Issues**
|
||
|
```bash
|
||
|
# Test MinIO connection
|
||
|
mc alias set local http://localhost:9000 access_key secret_key
|
||
|
mc ls local
|
||
|
```
|
||
|
|
||
|
3. **Queue Processing Stopped**
|
||
|
```bash
|
||
|
# Check Redis connection
|
||
|
redis-cli ping
|
||
|
|
||
|
# Check queue status
|
||
|
curl http://localhost:3002/health/detailed
|
||
|
```
|
||
|
|
||
|
4. **High Memory Usage**
|
||
|
```bash
|
||
|
# Check temp file cleanup
|
||
|
ls -la /tmp/seo-worker/
|
||
|
|
||
|
# Force cleanup
|
||
|
curl -X POST http://localhost:3002/admin/cleanup
|
||
|
```
|
||
|
|
||
|
### Debugging
|
||
|
|
||
|
Enable debug logging:
|
||
|
```env
|
||
|
LOG_LEVEL=debug
|
||
|
NODE_ENV=development
|
||
|
```
|
||
|
|
||
|
Monitor processing in real-time:
|
||
|
```bash
|
||
|
# Follow logs
|
||
|
docker logs -f seo-worker
|
||
|
|
||
|
# Monitor metrics
|
||
|
curl http://localhost:9090/metrics | grep seo_worker
|
||
|
```
|
||
|
|
||
|
## Development
|
||
|
|
||
|
### Project Structure
|
||
|
```
|
||
|
src/
|
||
|
├── config/ # Configuration and validation
|
||
|
├── vision/ # AI vision services
|
||
|
├── processors/ # BullMQ job processors
|
||
|
├── storage/ # File and cloud storage
|
||
|
├── queue/ # Queue management and tracking
|
||
|
├── security/ # Virus scanning and validation
|
||
|
├── database/ # Database integration
|
||
|
├── monitoring/ # Metrics and logging
|
||
|
└── health/ # Health check endpoints
|
||
|
```
|
||
|
|
||
|
### Testing
|
||
|
```bash
|
||
|
# Unit tests
|
||
|
npm test
|
||
|
|
||
|
# Integration tests
|
||
|
npm run test:e2e
|
||
|
|
||
|
# Coverage report
|
||
|
npm run test:cov
|
||
|
```
|
||
|
|
||
|
### Contributing
|
||
|
|
||
|
1. Fork the repository
|
||
|
2. Create a feature branch
|
||
|
3. Add comprehensive tests
|
||
|
4. Update documentation
|
||
|
5. Submit a pull request
|
||
|
|
||
|
## License
|
||
|
|
||
|
Proprietary - SEO Image Renamer Platform
|
||
|
|
||
|
## Support
|
||
|
|
||
|
For technical support and questions:
|
||
|
- Documentation: [Internal Wiki]
|
||
|
- Issues: [Project Board]
|
||
|
- Contact: engineering@seo-image-renamer.com
|