Skip to main content

Routine Flow

Documentation of the routine workflows and processes in the Hawkeye system.

Overview​

Hawkeye follows a structured routine flow for incident processing, from initial submission through prediction, classification, and response.

ML Model Deployment​

The ML model is deployed on BentoML Cloud:

Main Routine Flow​

1. Incident Submission Flow​

User/System Submission
    ↓
Input Validation
    ↓
Feature Extraction
    ↓
AI Prediction
    ↓
Result Processing
    ↓
Response/Notification

Detailed Steps​

async function processIncidentSubmission(incidentData: IncidentData) {
  // Step 1: Validate input
  const validation = validateIncidentInput(incidentData);
  if (!validation.valid) {
    return { error: validation.errors };
  }
  
  // Step 2: Extract features
  const features = await extractFeatures(incidentData);
  
  // Step 3: AI prediction (via BentoML Cloud)
  const prediction = await makePrediction(features);
  
  // Step 4: Process result
  const result = processPrediction(prediction);
  
  // Step 5: Store and notify
  await storeIncident(incidentData, result);
  await notifyStakeholders(result);
  
  return result;
}

2. Prediction Flow​

The routine flow for making predictions via BentoML Cloud:

import requests

BENTOML_API_URL = "https://failure-prediction-prod-0d460137.mt-guc1.bentoml.ai/predict"

def routine_prediction_flow(incident_data):
    """
    Standard routine for processing predictions via BentoML Cloud
    """
    # 1. Pre-processing
    validated_data = validate_incident_data(incident_data)
    
    # 2. Feature extraction
    features = extract_all_features(validated_data)
    
    # 3. Prepare request payload
    payload = {
        "incident_type": features.get('incident_type'),
        "severity": features.get('severity'),
        "description": validated_data.get('description'),
        "affected_users": features.get('affected_users', 0),
        "system_impact_score": features.get('system_impact_score', 0)
    }
    
    # 4. Call BentoML Cloud API
    try:
        response = requests.post(BENTOML_API_URL, json=payload, timeout=10)
        response.raise_for_status()
        api_result = response.json()
        
        # 5. Extract prediction from API response
        raw_prediction = api_result.get('prediction')
        probabilities = api_result.get('probabilities', {})
        confidence = api_result.get('confidence', 0.0)
        
    except requests.RequestException as e:
        # Fallback to local model if API fails
        processed_features = preprocess_features(features)
        raw_prediction = model.predict(processed_features)
        probabilities = model.predict_proba(processed_features)
        confidence = probabilities.max()
    
    # 6. Post-processing
    prediction_result = format_prediction(
        raw_prediction, 
        probabilities, 
        features,
        confidence
    )
    
    # 7. Decision making
    decision = make_decision(prediction_result)
    
    # 8. Logging and storage
    log_prediction(prediction_result, decision)
    store_prediction(prediction_result)
    
    return prediction_result, decision

3. Batch Processing Flow​

Routine for processing multiple incidents:

def batch_processing_routine(incidents):
    """
    Routine flow for batch incident processing
    """
    results = []
    
    # 1. Validate batch
    validated_incidents = [v for v in incidents if validate(v)]
    
    # 2. Extract features for all
    feature_matrix = extract_batch_features(validated_incidents)
    
    # 3. Preprocess batch
    processed_batch = preprocess_batch(feature_matrix)
    
    # 4. Batch prediction
    predictions = model.predict(processed_batch)
    probabilities = model.predict_proba(processed_batch)
    
    # 5. Process results
    for i, (pred, prob) in enumerate(zip(predictions, probabilities)):
        result = format_prediction(pred, prob, feature_matrix[i])
        results.append(result)
    
    # 6. Store batch results
    store_batch_results(results)
    
    return results

Workflow States​

Incident State Machine​

CREATED → VALIDATED → PROCESSING → PREDICTED → CLASSIFIED → RESOLVED
   ↓         ↓            ↓            ↓            ↓
  ERROR   ERROR        ERROR        ERROR        ERROR

State Transitions​

enum IncidentState {
  CREATED = 'created',
  VALIDATED = 'validated',
  PROCESSING = 'processing',
  PREDICTED = 'predicted',
  CLASSIFIED = 'classified',
  RESOLVED = 'resolved',
  ERROR = 'error'
}

function transitionState(current: IncidentState, action: string): IncidentState {
  const transitions: Record<IncidentState, Record<string, IncidentState>> = {
    [IncidentState.CREATED]: {
      'validate': IncidentState.VALIDATED,
      'error': IncidentState.ERROR
    },
    [IncidentState.VALIDATED]: {
      'process': IncidentState.PROCESSING,
      'error': IncidentState.ERROR
    },
    [IncidentState.PROCESSING]: {
      'predict': IncidentState.PREDICTED,
      'error': IncidentState.ERROR
    },
    [IncidentState.PREDICTED]: {
      'classify': IncidentState.CLASSIFIED,
      'error': IncidentState.ERROR
    },
    [IncidentState.CLASSIFIED]: {
      'resolve': IncidentState.RESOLVED,
      'error': IncidentState.ERROR
    }
  };
  
  return transitions[current]?.[action] || current;
}

Error Handling Flow​

Error Recovery Routine​

def error_handling_routine(error, context):
    """
    Standard error handling flow
    """
    # 1. Log error
    log_error(error, context)
    
    # 2. Classify error
    error_type = classify_error(error)
    
    # 3. Handle based on type
    if error_type == 'VALIDATION_ERROR':
        return handle_validation_error(error, context)
    elif error_type == 'MODEL_ERROR':
        return handle_model_error(error, context)
    elif error_type == 'DATABASE_ERROR':
        return handle_database_error(error, context)
    else:
        return handle_unknown_error(error, context)
    
    # 4. Notify if critical
    if is_critical_error(error):
        notify_administrators(error, context)
    
    # 5. Return fallback response
    return create_fallback_response(context)

Notification Flow​

Stakeholder Notification Routine​

def notification_routine(prediction_result):
    """
    Routine for notifying stakeholders
    """
    # 1. Determine notification recipients
    recipients = determine_recipients(prediction_result)
    
    # 2. Prepare notification content
    content = prepare_notification_content(prediction_result)
    
    # 3. Send notifications
    for recipient in recipients:
        if recipient['preference'] == 'email':
            send_email(recipient, content)
        elif recipient['preference'] == 'slack':
            send_slack_message(recipient, content)
        elif recipient['preference'] == 'webhook':
            send_webhook(recipient, content)
    
    # 4. Log notifications
    log_notifications(recipients, content)

Data Processing Flow​

Routine Data Processing​

def data_processing_routine(raw_data):
    """
    Standard data processing routine
    """
    # 1. Clean data
    cleaned = clean_data(raw_data)
    
    # 2. Transform data
    transformed = transform_data(cleaned)
    
    # 3. Enrich data
    enriched = enrich_data(transformed)
    
    # 4. Validate processed data
    validated = validate_processed_data(enriched)
    
    # 5. Store processed data
    store_processed_data(validated)
    
    return validated

Periodic Maintenance Flow​

Daily Maintenance Routine​

def daily_maintenance_routine():
    """
    Routine maintenance tasks
    """
    # 1. Clean old predictions
    cleanup_old_predictions(days=90)
    
    # 2. Update statistics
    update_prediction_statistics()
    
    # 3. Check model performance
    performance = check_model_performance()
    
    # 4. Generate reports
    generate_daily_report(performance)
    
    # 5. Backup data
    backup_prediction_data()

Integration Flow​

External System Integration Routine​

def integration_routine(external_data):
    """
    Routine for integrating with external systems
    """
    # 1. Receive external data
    received = receive_external_data(external_data)
    
    # 2. Transform to internal format
    transformed = transform_to_internal_format(received)
    
    # 3. Process through standard flow
    result = processIncidentSubmission(transformed)
    
    # 4. Send response back
    send_external_response(result)
    
    return result

Flow Diagrams​

Complete Incident Processing Flow​

┌─────────────┐
│   Incident  │
│  Submitted  │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Validation │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   Feature   │
│ Extraction  │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Preprocessing│
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   AI Model  │
│  Prediction │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Post-process│
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Decision   │
│   Making    │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Storage &  │
│ Notification│
└─────────────┘

Next Steps​