Operations Runbook

This guide covers operational procedures for running Armada in production environments.

Pre-Deployment Checklist

Environment Requirements

Jira Cloud instance (Jira Software, Jira Service Management)
Node.js 18+ (for local development)
Docker (for containerized deployment)
Atlassian Forge CLI installed

Configuration Verification

# Verify Forge credentials
forge settings list

# Check app installation status
forge app list

# Verify environment configuration
forge variables list --environment production

Security Requirements

Jira admin permissions for app installation
API token generated from Atlassian account
Storage quotas reviewed and allocated
User permissions configured appropriately

Deployment Procedures

Development Environment

# Deploy to development
forge deploy --environment development

# Verify deployment
forge logs --environment development

# Install on test site
forge install --upgrade --site your-site.atlassian.net --product Jira

Production Deployment

Pre-deployment validation

npm run validate    # Run typecheck, lint, and tests
npm run build        # Build all resources

Deploy to production
Terminal window
```
forge deploy --environment production
```

Post-deployment verification

# Check app status
forge app status

# View production logs
forge logs --environment production

Update Marketplace listing (if version changed)
- Increment version in manifest.yml
- Update release notes
- Publish to marketplace

Monitoring & Observability

Key Metrics to Track

Metric	Description	Alert Threshold
Campaign success rate	% of campaigns completing successfully	< 95%
Average campaign duration	Time from launch to completion	> 72 hours
Auto-nudge effectiveness	% of nudged issues updated	< 50%
API error rate	Failed Jira API calls	> 1%
Storage utilization	Campaign state storage usage	> 80%

Log Analysis

# View recent logs
forge logs --environment production --tail 100

# Filter by severity
forge logs --environment production --filter error

# Search for specific campaign
forge logs --environment production --search PROJ-123

Health Checks

# Check app health
curl https://your-site.atlassian.net/rest/forge-Health/1.0/health

# Expected response
{
  "status": "UP",
  "components": {
    "storage": "UP",
    "jiraApi": "UP",
    "compassApi": "UP"
  }
}

Incident Response

High Error Rate

Immediate actions

# View detailed errors
forge logs --environment production --filter error --tail 500

# Check Jira API status
# Verify app permissions

Common causes
- Jira API rate limiting → Implement exponential backoff
- Permission changes → Re-install app
- Storage quota exceeded → Archive old campaigns
Resolution steps
- Clear campaign state cache
- Restart affected workflows
- Notify affected users

Storage Issues

# Check storage utilization
# Navigate to Fleet > Settings in Armada panel

# Archive old campaigns
# 1. Export campaign data to JSON
# 2. Delete old campaign state
# 3. Keep parent issue for audit trail

Permission Problems

Re-install the app:

forge install --upgrade --site your-site.atlassian.net

Verify scopes in manifest.yml match required permissions
Check user has appropriate Jira admin rights

Backup & Recovery

Campaign Data Backup

Campaign data is stored in:

Jira issue properties (campaign state)
Forge Storage (fleet config, templates)

# Export fleet configuration
# Available in Fleet > Settings > Export

# Export campaign data
# Available in Campaign Status > Export

Recovery Procedures

Fleet configuration loss
- Import from previously exported JSON
- Recreate teams and settings from documentation
Campaign state corruption
- Delete corrupted state via Armada UI
- Re-sync by fetching current issue status
- Lost history cannot be recovered (stateless by design)

Capacity Planning

Storage Limits

Component	Limit	Scaling Strategy
Campaign children	400 per campaign	Use sub-campaigns
Fleet teams	50 per installation	Multiple fleets
Mission templates	100 per installation	Archive old templates
Storage total	32KB per property	Chunked storage enabled

Rate Limiting

Jira REST API: 100 requests/minute (adjustable)
Batch size: 10 issues per request
Retry with exponential backoff: 1s, 2s, 4s, 8s

Maintenance Windows

Recommended Schedule

Weekly: Log review and cleanup
Monthly: Capacity assessment
Quarterly: Security audit and updates

Update Procedures

Review changelog for breaking changes
Test in development environment
Schedule maintenance window
Deploy during low-usage period
Verify all features working
Monitor for 24 hours post-update

Support Contacts

Role	Contact	Response Time
Technical Support	[email protected]	24 hours
Security Issues	[email protected]	4 hours
Enterprise Support	[email protected]	1 hour
Sales	[email protected]	24 hours

Escalation Path

Level 1: Check documentation and logs
Level 2: Submit support ticket
Level 3: Contact account manager
Level 4: Emergency escalation to engineering lead