Regex for Logs and Monitoring – Parse, Filter, and Alert

Log analysis is crucial for maintaining healthy systems and applications. Regular expressions provide powerful tools for parsing, filtering, and extracting meaningful data from log files. This comprehensive guide covers regex patterns for common log formats, monitoring strategies, alerting systems, and performance optimization techniques.

Table of Contents

1. Log Parsing Fundamentals
2. Common Log Format Patterns
3. Application Log Patterns
4. System Log Patterns
5. Monitoring and Alerting Strategies
6. Performance Optimization for Log Processing
7. Real-time Log Processing
8. Security Log Analysis
9. Troubleshooting and Debug Patterns

Log Parsing Fundamentals

Understanding Log Structure

Most log entries follow a predictable structure that can be parsed with regex:

Basic Log Entry Components

• Timestamp: When the event occurred
• Level: Severity or importance (DEBUG, INFO, WARN, ERROR, FATAL)
• Source: Component, module, or service that generated the log
• Message: The actual log content
• Context: Additional metadata (user ID, request ID, etc.)

Generic Log Parser

class LogParser {
  constructor() {
    this.patterns = {
      // ISO 8601 timestamp
      timestamp: /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?(?:Z|[+-]\d{2}:\d{2})/,
      
      // Common timestamp formats
      timestamp_syslog: /[A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}/,
      timestamp_apache: /\d{2}\/[A-Z][a-z]{2}\/\d{4}:\d{2}:\d{2}:\d{2}\s+[+-]\d{4}/,
      timestamp_nginx: /\d{2}\/[A-Z][a-z]{2}\/\d{4}:\d{2}:\d{2}:\d{2}\s+[+-]\d{4}/,
      
      // Log levels
      level: /(DEBUG|INFO|WARN|WARNING|ERROR|FATAL|CRITICAL|TRACE)/i,
      
      // IP addresses
      ipv4: /\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b/,
      ipv6: /\b(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}\b/,
      
      // HTTP status codes
      httpStatus: /\s([1-5]\d{2})\s/,
      
      // Request IDs and correlation IDs
      requestId: /(?:request_id|req_id|correlation_id)[=:\s]+([a-zA-Z0-9-]{8,})/i,
      
      // User IDs
      userId: /(?:user_id|uid)[=:\s]+([a-zA-Z0-9_.-]+)/i,
      
      // Response times
      responseTime: /(?:response_time|duration|elapsed)[=:\s]+(\d+(?:\.\d+)?)\s*(?:ms|s|μs|ns)?/i
    };
  }
  
  parseLogEntry(logLine) {
    const parsed = {
      original: logLine,
      timestamp: null,
      level: null,
      message: null,
      metadata: {}
    };
    
    // Extract timestamp
    const timestampMatch = logLine.match(this.patterns.timestamp) ||
                          logLine.match(this.patterns.timestamp_syslog) ||
                          logLine.match(this.patterns.timestamp_apache);
    if (timestampMatch) {
      parsed.timestamp = timestampMatch[0];
    }
    
    // Extract log level
    const levelMatch = logLine.match(this.patterns.level);
    if (levelMatch) {
      parsed.level = levelMatch[1].toUpperCase();
    }
    
    // Extract IP addresses
    const ipMatch = logLine.match(this.patterns.ipv4) || logLine.match(this.patterns.ipv6);
    if (ipMatch) {
      parsed.metadata.ip = ipMatch[0];
    }
    
    // Extract HTTP status code
    const statusMatch = logLine.match(this.patterns.httpStatus);
    if (statusMatch) {
      parsed.metadata.status = parseInt(statusMatch[1]);
    }
    
    // Extract request ID
    const requestIdMatch = logLine.match(this.patterns.requestId);
    if (requestIdMatch) {
      parsed.metadata.requestId = requestIdMatch[1];
    }
    
    // Extract user ID
    const userIdMatch = logLine.match(this.patterns.userId);
    if (userIdMatch) {
      parsed.metadata.userId = userIdMatch[1];
    }
    
    // Extract response time
    const responseTimeMatch = logLine.match(this.patterns.responseTime);
    if (responseTimeMatch) {
      parsed.metadata.responseTime = parseFloat(responseTimeMatch[1]);
    }
    
    // Extract remaining message (remove parsed components)
    let message = logLine;
    if (parsed.timestamp) {
      message = message.replace(parsed.timestamp, '').trim();
    }
    if (parsed.level) {
      message = message.replace(new RegExp(`\\b${parsed.level}\\b`, 'i'), '').trim();
    }
    
    parsed.message = message;
    
    return parsed;
  }
  
  parseMultipleLines(logText) {
    const lines = logText.split('\n').filter(line => line.trim());
    return lines.map(line => this.parseLogEntry(line));
  }
}

// Usage example
const parser = new LogParser();
const logLine = '2024-01-15T10:30:45.123Z INFO [web-server] Request processed: method=GET path=/api/users status=200 duration=45ms user_id=user123';
const parsed = parser.parseLogEntry(logLine);
console.log(parsed);

Common Log Format Patterns

Apache Access Log Patterns

class ApacheLogParser {
  constructor() {
    // Apache Common Log Format (CLF)
    this.clfPattern = /^([\d\.]+)\s+(\S+)\s+(\S+)\s+\[([^\]]+)\]\s+"([^"]+)"\s+(\d{3})\s+(\d+|-)$/;
    
    // Apache Combined Log Format
    this.combinedPattern = /^([\d\.]+)\s+(\S+)\s+(\S+)\s+\[([^\]]+)\]\s+"([^"]+)"\s+(\d{3})\s+(\d+|-)\s+"([^"]+)"\s+"([^"]+)"$/;
    
    // Apache Error Log
    this.errorLogPattern = /^\[([^\]]+)\]\s+\[([^\]]+)\]\s+\[([^\]]+)\]\s+(.*)$/;
  }
  
  parseCommonLogFormat(logLine) {
    const match = logLine.match(this.clfPattern);
    if (!match) return null;
    
    return {
      ip: match[1],
      identity: match[2] === '-' ? null : match[2],
      user: match[3] === '-' ? null : match[3],
      timestamp: match[4],
      request: match[5],
      status: parseInt(match[6]),
      size: match[7] === '-' ? 0 : parseInt(match[7]),
      method: this.extractMethod(match[5]),
      path: this.extractPath(match[5]),
      protocol: this.extractProtocol(match[5])
    };
  }
  
  parseCombinedLogFormat(logLine) {
    const match = logLine.match(this.combinedPattern);
    if (!match) return null;
    
    const parsed = this.parseCommonLogFormat(logLine);
    if (parsed) {
      parsed.referer = match[8] === '-' ? null : match[8];
      parsed.userAgent = match[9];
    }
    
    return parsed;
  }
  
  parseErrorLog(logLine) {
    const match = logLine.match(this.errorLogPattern);
    if (!match) return null;
    
    return {
      timestamp: match[1],
      level: match[2],
      client: match[3],
      message: match[4]
    };
  }
  
  extractMethod(request) {
    const methodMatch = request.match(/^([A-Z]+)\s/);
    return methodMatch ? methodMatch[1] : null;
  }
  
  extractPath(request) {
    const pathMatch = request.match(/^[A-Z]+\s+(\S+)/);
    return pathMatch ? pathMatch[1] : null;
  }
  
  extractProtocol(request) {
    const protocolMatch = request.match(/\s+(HTTP\/[\d\.]+)$/);
    return protocolMatch ? protocolMatch[1] : null;
  }
}

Nginx Log Patterns

class NginxLogParser {
  constructor() {
    // Standard nginx access log format
    this.accessLogPattern = /^([\d\.]+)\s+-\s+([^\s]+)\s+\[([^\]]+)\]\s+"([^"]+)"\s+(\d{3})\s+(\d+)\s+"([^"]+)"\s+"([^"]+)"(?:\s+"([^"]+)")?$/;
    
    // Custom nginx log format with additional fields
    this.customLogPattern = /^([\d\.]+)\s+-\s+([^\s]+)\s+\[([^\]]+)\]\s+"([^"]+)"\s+(\d{3})\s+(\d+)\s+"([^"]+)"\s+"([^"]+)"\s+rt=([\d\.]+)\s+uct="([^"]+)"\s+uht="([^"]+)"\s+urt="([^"]+)"$/;
    
    // Nginx error log
    this.errorLogPattern = /^(\d{4}\/\d{2}\/\d{2}\s+\d{2}:\d{2}:\d{2})\s+\[([^\]]+)\]\s+(\d+)#(\d+):\s+(.*)$/;
  }
  
  parseAccessLog(logLine) {
    let match = logLine.match(this.customLogPattern);
    if (match) {
      return {
        ip: match[1],
        user: match[2] === '-' ? null : match[2],
        timestamp: match[3],
        request: match[4],
        status: parseInt(match[5]),
        bytes: parseInt(match[6]),
        referer: match[7] === '-' ? null : match[7],
        userAgent: match[8],
        requestTime: parseFloat(match[9]),
        upstreamConnectTime: match[10],
        upstreamHeaderTime: match[11],
        upstreamResponseTime: match[12]
      };
    }
    
    match = logLine.match(this.accessLogPattern);
    if (match) {
      return {
        ip: match[1],
        user: match[2] === '-' ? null : match[2],
        timestamp: match[3],
        request: match[4],
        status: parseInt(match[5]),
        bytes: parseInt(match[6]),
        referer: match[7] === '-' ? null : match[7],
        userAgent: match[8],
        realIp: match[9] || null
      };
    }
    
    return null;
  }
  
  parseErrorLog(logLine) {
    const match = logLine.match(this.errorLogPattern);
    if (!match) return null;
    
    return {
      timestamp: match[1],
      level: match[2],
      pid: parseInt(match[3]),
      tid: parseInt(match[4]),
      message: match[5]
    };
  }
}

Application Framework Log Patterns

class ApplicationLogParser {
  constructor() {
    this.patterns = {
      // Spring Boot / Logback format
      springBoot: /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d{3})\s+([A-Z]+)\s+(\d+)\s+---\s+\[([^\]]+)\]\s+([^\s]+)\s*:\s+(.*)$/,
      
      // Node.js / Winston format
      winston: /^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\s+([a-z]+):\s+(.*)(?:\s+\{(.*)\})?$/,
      
      // Python / logging format
      pythonLogging: /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2},\d{3})\s+-\s+([^\s]+)\s+-\s+([A-Z]+)\s+-\s+(.*)$/,
      
      // Ruby on Rails format
      rails: /^([A-Z])\s*,\s*\[([^\]]+)#(\d+)\]\s+([A-Z]+)\s+--\s*:\s*(.*)$/,
      
      // .NET / Serilog format
      serilog: /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d{3})\s+\[([A-Z]+)\]\s+(.*)$/,
      
      // Docker container logs
      docker: /^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+stdout\s+F\s+(.*)$/,
      
      // Kubernetes logs
      kubernetes: /^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+([a-zA-Z0-9-]+)\s+([a-zA-Z0-9-]+)\s+(.*)$/
    };
  }
  
  parseSpringBootLog(logLine) {
    const match = logLine.match(this.patterns.springBoot);
    if (!match) return null;
    
    return {
      timestamp: match[1],
      level: match[2],
      pid: parseInt(match[3]),
      thread: match[4],
      logger: match[5],
      message: match[6]
    };
  }
  
  parseWinstonLog(logLine) {
    const match = logLine.match(this.patterns.winston);
    if (!match) return null;
    
    let metadata = null;
    if (match[4]) {
      try {
        metadata = JSON.parse('{' + match[4] + '}');
      } catch (e) {
        metadata = { raw: match[4] };
      }
    }
    
    return {
      timestamp: match[1],
      level: match[2],
      message: match[3],
      metadata
    };
  }
  
  parsePythonLog(logLine) {
    const match = logLine.match(this.patterns.pythonLogging);
    if (!match) return null;
    
    return {
      timestamp: match[1],
      logger: match[2],
      level: match[3],
      message: match[4]
    };
  }
  
  parseDockerLog(logLine) {
    const match = logLine.match(this.patterns.docker);
    if (!match) return null;
    
    return {
      timestamp: match[1],
      stream: 'stdout',
      message: match[2]
    };
  }
  
  parseKubernetesLog(logLine) {
    const match = logLine.match(this.patterns.kubernetes);
    if (!match) return null;
    
    return {
      timestamp: match[1],
      namespace: match[2],
      pod: match[3],
      message: match[4]
    };
  }
  
  autoDetectAndParse(logLine) {
    // Try each format until one matches
    const parsers = [
      { name: 'springBoot', fn: this.parseSpringBootLog.bind(this) },
      { name: 'winston', fn: this.parseWinstonLog.bind(this) },
      { name: 'python', fn: this.parsePythonLog.bind(this) },
      { name: 'docker', fn: this.parseDockerLog.bind(this) },
      { name: 'kubernetes', fn: this.parseKubernetesLog.bind(this) }
    ];
    
    for (const parser of parsers) {
      const result = parser.fn(logLine);
      if (result) {
        result.format = parser.name;
        return result;
      }
    }
    
    return { format: 'unknown', message: logLine };
  }
}

Application Log Patterns

Error Detection and Classification

class ErrorDetector {
  constructor() {
    this.errorPatterns = {
      // Application errors
      exception: /(?:Exception|Error)\s*(?:in|at)?\s*([^\n]+)/i,
      stackTrace: /^\s*at\s+([^\(]+)\(([^\)]+)\)/,
      
      // Database errors
      dbConnection: /(?:connection|database).*(?:refused|timeout|failed|error)/i,
      sqlError: /SQL.*(?:syntax|error|exception)/i,
      deadlock: /deadlock|lock\s+timeout/i,
      
      // HTTP errors
      httpError: /HTTP\s+(4\d{2}|5\d{2})/,
      timeoutError: /timeout|timed\s+out/i,
      
      // Memory issues
      outOfMemory: /out\s+of\s+memory|OutOfMemoryError/i,
      memoryLeak: /memory\s+leak|heap\s+space/i,
      
      // Security issues
      authError: /(?:authentication|authorization).*(?:failed|denied|error)/i,
      accessDenied: /access\s+denied|permission\s+denied/i,
      
      // Performance issues
      slowQuery: /slow\s+query|query\s+exceeded/i,
      highLatency: /(?:response\s+time|latency).*(?:high|exceeded|\d{4,}ms)/i,
      
      // Infrastructure issues
      diskSpace: /no\s+space\s+left|disk\s+full/i,
      networkError: /network.*(?:unreachable|timeout|error)/i
    };
    
    this.severityLevels = {
      CRITICAL: ['outOfMemory', 'dbConnection', 'diskSpace'],
      HIGH: ['exception', 'sqlError', 'authError', 'accessDenied'],
      MEDIUM: ['httpError', 'timeoutError', 'slowQuery'],
      LOW: ['stackTrace', 'highLatency', 'networkError']
    };
  }
  
  detectErrors(logLine) {
    const detectedIssues = [];
    
    for (const [type, pattern] of Object.entries(this.errorPatterns)) {
      const match = logLine.match(pattern);
      if (match) {
        const severity = this.getSeverity(type);
        detectedIssues.push({
          type,
          severity,
          match: match[0],
          details: match[1] || null,
          pattern: pattern.source
        });
      }
    }
    
    return detectedIssues;
  }
  
  getSeverity(errorType) {
    for (const [severity, types] of Object.entries(this.severityLevels)) {
      if (types.includes(errorType)) {
        return severity;
      }
    }
    return 'UNKNOWN';
  }
  
  extractStackTrace(logText) {
    const stackTracePattern = /(?:Exception|Error).*?\n((?:\s*at\s+.*\n?)+)/g;
    const matches = [];
    let match;
    
    while ((match = stackTracePattern.exec(logText)) !== null) {
      const stackLines = match[1].trim().split('\n');
      const parsedStack = stackLines.map(line => {
        const lineMatch = line.match(/^\s*at\s+([^\(]+)\(([^\)]+)\)/);
        if (lineMatch) {
          return {
            method: lineMatch[1].trim(),
            location: lineMatch[2].trim(),
            raw: line.trim()
          };
        }
        return { raw: line.trim() };
      });
      
      matches.push({
        exception: match[0].split('\n')[0],
        stackTrace: parsedStack
      });
    }
    
    return matches;
  }
  
  categorizeError(errorDescription) {
    const categories = {
      'Frontend': /(?:javascript|js|frontend|client-side|browser)/i,
      'Backend': /(?:server|backend|api|service)/i,
      'Database': /(?:sql|database|db|query|table)/i,
      'Network': /(?:network|connection|socket|http|https)/i,
      'Authentication': /(?:auth|login|token|session|credential)/i,
      'Validation': /(?:validation|invalid|malformed|format)/i,
      'Performance': /(?:slow|timeout|latency|performance|memory)/i,
      'Security': /(?:security|attack|intrusion|exploit)/i
    };
    
    for (const [category, pattern] of Object.entries(categories)) {
      if (pattern.test(errorDescription)) {
        return category;
      }
    }
    
    return 'General';
  }
}

Performance Monitoring Patterns

class PerformanceMonitor {
  constructor() {
    this.patterns = {
      // Response time patterns
      responseTime: /(?:response_time|duration|elapsed|took)[:=]?\s*(\d+(?:\.\d+)?)\s*(ms|s|μs|ns)?/i,
      
      // Database query performance
      queryTime: /(?:query|sql).*?(?:duration|time|took)[:=]?\s*(\d+(?:\.\d+)?)\s*(ms|s)?/i,
      
      // Memory usage
      memoryUsage: /(?:memory|heap|ram).*?(?:used|usage|size)[:=]?\s*(\d+(?:\.\d+)?)\s*(MB|GB|KB|B)?/i,
      
      // CPU usage
      cpuUsage: /cpu.*?(?:usage|load)[:=]?\s*(\d+(?:\.\d+)?)%?/i,
      
      // Request rate
      requestRate: /(?:requests?|req).*?(?:per|\/)\s*(?:second|sec|minute|min)[:=]?\s*(\d+(?:\.\d+)?)/i,
      
      // Cache hit/miss
      cacheHit: /cache.*?hit.*?(?:rate)?[:=]?\s*(\d+(?:\.\d+)?)%?/i,
      cacheMiss: /cache.*?miss.*?(?:rate)?[:=]?\s*(\d+(?:\.\d+)?)%?/i,
      
      // Thread pool usage
      threadPool: /thread.*?pool.*?(?:size|active|used)[:=]?\s*(\d+)/i,
      
      // Garbage collection
      gcTime: /gc.*?(?:time|duration)[:=]?\s*(\d+(?:\.\d+)?)\s*(ms|s)?/i,
      
      // Disk I/O
      diskIO: /disk.*?(?:read|write|io).*?(?:rate|speed)[:=]?\s*(\d+(?:\.\d+)?)\s*(MB\/s|KB\/s|B\/s)?/i
    };
    
    this.thresholds = {
      responseTime: { warning: 1000, critical: 5000 }, // milliseconds
      queryTime: { warning: 500, critical: 2000 }, // milliseconds
      memoryUsage: { warning: 80, critical: 95 }, // percentage
      cpuUsage: { warning: 70, critical: 90 }, // percentage
      cacheHit: { warning: 80, critical: 60 }, // percentage (lower is worse)
      threadPool: { warning: 80, critical: 95 } // percentage of max
    };
  }
  
  extractPerformanceMetrics(logLine) {
    const metrics = {};
    
    for (const [metricName, pattern] of Object.entries(this.patterns)) {
      const match = logLine.match(pattern);
      if (match) {
        const value = parseFloat(match[1]);
        const unit = match[2] || 'unknown';
        
        metrics[metricName] = {
          value: this.normalizeValue(value, unit, metricName),
          unit: this.getStandardUnit(metricName),
          raw: match[0]
        };
      }
    }
    
    return metrics;
  }
  
  normalizeValue(value, unit, metricName) {
    // Normalize to standard units
    const normalizations = {
      responseTime: {
        's': value * 1000,
        'ms': value,
        'μs': value / 1000,
        'ns': value / 1000000
      },
      memoryUsage: {
        'GB': value * 1024,
        'MB': value,
        'KB': value / 1024,
        'B': value / (1024 * 1024)
      }
    };
    
    const normalizer = normalizations[metricName];
    if (normalizer && normalizer[unit]) {
      return normalizer[unit];
    }
    
    return value;
  }
  
  getStandardUnit(metricName) {
    const units = {
      responseTime: 'ms',
      queryTime: 'ms',
      memoryUsage: 'MB',
      cpuUsage: '%',
      requestRate: 'req/s',
      cacheHit: '%',
      cacheMiss: '%',
      threadPool: 'count',
      gcTime: 'ms',
      diskIO: 'MB/s'
    };
    
    return units[metricName] || 'unknown';
  }
  
  checkThresholds(metrics) {
    const alerts = [];
    
    for (const [metricName, metricData] of Object.entries(metrics)) {
      const threshold = this.thresholds[metricName];
      if (!threshold) continue;
      
      const value = metricData.value;
      
      if (value >= threshold.critical) {
        alerts.push({
          metric: metricName,
          level: 'CRITICAL',
          value,
          threshold: threshold.critical,
          message: `${metricName} is critically high: ${value}${metricData.unit}`
        });
      } else if (value >= threshold.warning) {
        alerts.push({
          metric: metricName,
          level: 'WARNING',
          value,
          threshold: threshold.warning,
          message: `${metricName} is above warning threshold: ${value}${metricData.unit}`
        });
      }
      
      // Special case for cache hit rate (lower is worse)
      if (metricName === 'cacheHit') {
        if (value <= threshold.critical) {
          alerts.push({
            metric: metricName,
            level: 'CRITICAL',
            value,
            threshold: threshold.critical,
            message: `Cache hit rate is critically low: ${value}%`
          });
        } else if (value <= threshold.warning) {
          alerts.push({
            metric: metricName,
            level: 'WARNING',
            value,
            threshold: threshold.warning,
            message: `Cache hit rate is below optimal: ${value}%`
          });
        }
      }
    }
    
    return alerts;
  }
  
  generatePerformanceReport(logLines) {
    const metrics = {
      responseTime: [],
      queryTime: [],
      memoryUsage: [],
      cpuUsage: []
    };
    
    logLines.forEach(line => {
      const extracted = this.extractPerformanceMetrics(line);
      
      Object.keys(metrics).forEach(metric => {
        if (extracted[metric]) {
          metrics[metric].push(extracted[metric].value);
        }
      });
    });
    
    const report = {};
    
    Object.entries(metrics).forEach(([metric, values]) => {
      if (values.length > 0) {
        const sorted = values.sort((a, b) => a - b);
        report[metric] = {
          count: values.length,
          min: Math.min(...values),
          max: Math.max(...values),
          avg: values.reduce((a, b) => a + b) / values.length,
          median: sorted[Math.floor(sorted.length / 2)],
          p95: sorted[Math.floor(sorted.length * 0.95)],
          p99: sorted[Math.floor(sorted.length * 0.99)]
        };
      }
    });
    
    return report;
  }
}

System Log Patterns

System Event Detection

class SystemLogAnalyzer {
  constructor() {
    this.patterns = {
      // System startup/shutdown
      systemBoot: /kernel.*?booting|system\s+(?:starting|booted)/i,
      systemShutdown: /system\s+(?:shutdown|halt|reboot)/i,
      
      // Service management
      serviceStart: /(?:service|daemon).*?(?:started|starting|active)/i,
      serviceStop: /(?:service|daemon).*?(?:stopped|stopping|inactive)/i,
      serviceFailed: /(?:service|daemon).*?(?:failed|error|crashed)/i,
      
      // Security events
      loginSuccess: /(?:login|authentication).*?(?:successful|succeeded|accepted)/i,
      loginFailure: /(?:login|authentication).*?(?:failed|denied|rejected)/i,
      privilegeEscalation: /sudo.*?(?:COMMAND|executed)/i,
      
      // Network events
      connectionEstablished: /connection.*?established/i,
      connectionRefused: /connection.*?refused/i,
      networkInterface: /(?:interface|network).*?(?:up|down|link)/i,
      
      // Storage events
      diskError: /disk.*?(?:error|failure|bad\s+sector)/i,
      filesystemError: /(?:filesystem|fs).*?(?:error|corruption|check)/i,
      mountError: /mount.*?(?:failed|error)/i,
      
      // Process events
      processStart: /process.*?(?:started|spawned|forked)/i,
      processExit: /process.*?(?:exited|terminated|killed)/i,
      processSegfault: /segmentation\s+fault|segfault/i,
      
      // Memory events
      oomKiller: /Out\s+of\s+memory.*?killed|oom-killer/i,
      swapUsage: /swap.*?(?:usage|full|space)/i,
      
      // Kernel events
      kernelPanic: /kernel\s+panic|panic.*?kernel/i,
      kernelOops: /kernel.*?oops|oops.*?kernel/i,
      hardwareError: /hardware.*?error|mce.*?error/i
    };
    
    this.eventSeverity = {
      CRITICAL: ['kernelPanic', 'kernelOops', 'hardwareError', 'oomKiller'],
      HIGH: ['serviceFailed', 'diskError', 'filesystemError', 'processSegfault'],
      MEDIUM: ['loginFailure', 'connectionRefused', 'mountError', 'swapUsage'],
      LOW: ['serviceStart', 'serviceStop', 'loginSuccess', 'processStart', 'processExit'],
      INFO: ['systemBoot', 'systemShutdown', 'connectionEstablished', 'networkInterface']
    };
  }
  
  analyzeSystemEvent(logLine) {
    const events = [];
    
    for (const [eventType, pattern] of Object.entries(this.patterns)) {
      const match = logLine.match(pattern);
      if (match) {
        const severity = this.getEventSeverity(eventType);
        events.push({
          type: eventType,
          severity,
          timestamp: this.extractTimestamp(logLine),
          details: match[0],
          category: this.categorizeEvent(eventType)
        });
      }
    }
    
    return events;
  }
  
  getEventSeverity(eventType) {
    for (const [severity, types] of Object.entries(this.eventSeverity)) {
      if (types.includes(eventType)) {
        return severity;
      }
    }
    return 'UNKNOWN';
  }
  
  categorizeEvent(eventType) {
    const categories = {
      System: ['systemBoot', 'systemShutdown', 'kernelPanic', 'kernelOops'],
      Service: ['serviceStart', 'serviceStop', 'serviceFailed'],
      Security: ['loginSuccess', 'loginFailure', 'privilegeEscalation'],
      Network: ['connectionEstablished', 'connectionRefused', 'networkInterface'],
      Storage: ['diskError', 'filesystemError', 'mountError'],
      Process: ['processStart', 'processExit', 'processSegfault'],
      Memory: ['oomKiller', 'swapUsage'],
      Hardware: ['hardwareError']
    };
    
    for (const [category, types] of Object.entries(categories)) {
      if (types.includes(eventType)) {
        return category;
      }
    }
    
    return 'General';
  }
  
  extractTimestamp(logLine) {
    const timestampPatterns = [
      /^([A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})/,
      /(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?(?:Z|[+-]\d{2}:\d{2})?)/,
      /(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})/
    ];
    
    for (const pattern of timestampPatterns) {
      const match = logLine.match(pattern);
      if (match) {
        return match[1];
      }
    }
    
    return null;
  }
  
  detectSystemAnomalies(logLines) {
    const anomalies = [];
    const eventCounts = {};
    const timeWindows = {};
    
    logLines.forEach(line => {
      const events = this.analyzeSystemEvent(line);
      
      events.forEach(event => {
        // Count event types
        eventCounts[event.type] = (eventCounts[event.type] || 0) + 1;
        
        // Check for rapid repeated events (potential issue)
        const eventKey = `${event.type}_${event.severity}`;
        if (!timeWindows[eventKey]) {
          timeWindows[eventKey] = [];
        }
        timeWindows[eventKey].push(Date.now());
        
        // Keep only events from last 5 minutes
        const fiveMinutesAgo = Date.now() - (5 * 60 * 1000);
        timeWindows[eventKey] = timeWindows[eventKey].filter(time => time > fiveMinutesAgo);
        
        // Alert if too many events in short period
        if (timeWindows[eventKey].length > 10 && event.severity === 'HIGH') {
          anomalies.push({
            type: 'RAPID_REPEAT',
            eventType: event.type,
            count: timeWindows[eventKey].length,
            timeframe: '5 minutes',
            severity: 'CRITICAL'
          });
        }
      });
    });
    
    // Detect unusual event patterns
    const totalEvents = Object.values(eventCounts).reduce((a, b) => a + b, 0);
    
    Object.entries(eventCounts).forEach(([eventType, count]) => {
      const percentage = (count / totalEvents) * 100;
      
      // Alert if high-severity events are more than 5% of total
      const severity = this.getEventSeverity(eventType);
      if (severity === 'HIGH' && percentage > 5) {
        anomalies.push({
          type: 'HIGH_ERROR_RATE',
          eventType,
          percentage: percentage.toFixed(2),
          count,
          severity: 'HIGH'
        });
      }
    });
    
    return anomalies;
  }
}

Monitoring and Alerting Strategies

Real-time Alert System

class LogAlertSystem {
  constructor(config = {}) {
    this.config = {
      alertThresholds: {
        error_rate: 5, // errors per minute
        response_time_p95: 2000, // milliseconds
        memory_usage: 80, // percentage
        disk_usage: 85, // percentage
        failed_logins: 10 // per 5 minutes
      },
      notificationChannels: config.notificationChannels || [],
      timeWindows: {
        short: 60 * 1000, // 1 minute
        medium: 5 * 60 * 1000, // 5 minutes
        long: 30 * 60 * 1000 // 30 minutes
      },
      ...config
    };
    
    this.alertHistory = new Map();
    this.metrics = new Map();
    this.ruleEngine = new AlertRuleEngine();
  }
  
  processLogEntry(logEntry) {
    const alerts = [];
    
    // Extract metrics
    const metrics = this.extractMetrics(logEntry);
    this.updateMetrics(metrics);
    
    // Check for immediate alerts
    const immediateAlerts = this.checkImmediateAlerts(logEntry);
    alerts.push(...immediateAlerts);
    
    // Check threshold-based alerts
    const thresholdAlerts = this.checkThresholdAlerts();
    alerts.push(...thresholdAlerts);
    
    // Check pattern-based alerts
    const patternAlerts = this.checkPatternAlerts(logEntry);
    alerts.push(...patternAlerts);
    
    // Process and send alerts
    alerts.forEach(alert => this.handleAlert(alert));
    
    return alerts;
  }
  
  checkImmediateAlerts(logEntry) {
    const alerts = [];
    const immediateRules = {
      critical_error: {
        pattern: /(?:CRITICAL|FATAL|PANIC|EMERGENCY)/i,
        severity: 'CRITICAL',
        description: 'Critical error detected'
      },
      
      security_breach: {
        pattern: /(?:security\s+breach|unauthorized\s+access|intrusion\s+detected)/i,
        severity: 'CRITICAL',
        description: 'Potential security breach'
      },
      
      service_down: {
        pattern: /(?:service\s+(?:down|failed|crashed|unavailable))/i,
        severity: 'HIGH',
        description: 'Service availability issue'
      },
      
      database_connection_lost: {
        pattern: /(?:database.*connection.*lost|db.*connection.*failed)/i,
        severity: 'HIGH',
        description: 'Database connectivity issue'
      },
      
      out_of_memory: {
        pattern: /(?:out\s+of\s+memory|OutOfMemoryError|OOM\s+killed)/i,
        severity: 'HIGH',
        description: 'Memory exhaustion detected'
      }
    };
    
    Object.entries(immediateRules).forEach(([ruleName, rule]) => {
      if (rule.pattern.test(logEntry.message)) {
        alerts.push({
          type: 'IMMEDIATE',
          rule: ruleName,
          severity: rule.severity,
          description: rule.description,
          timestamp: logEntry.timestamp,
          details: logEntry.message,
          source: logEntry.source || 'unknown'
        });
      }
    });
    
    return alerts;
  }
  
  checkThresholdAlerts() {
    const alerts = [];
    const now = Date.now();
    
    // Check error rate
    const errorCount = this.getMetricInWindow('errors', this.config.timeWindows.short);
    if (errorCount > this.config.alertThresholds.error_rate) {
      alerts.push({
        type: 'THRESHOLD',
        metric: 'error_rate',
        value: errorCount,
        threshold: this.config.alertThresholds.error_rate,
        severity: 'HIGH',
        description: `Error rate exceeded: ${errorCount}/min`
      });
    }
    
    // Check response time
    const responseTimes = this.getMetricInWindow('response_time', this.config.timeWindows.medium);
    if (responseTimes.length > 0) {
      const p95 = this.calculatePercentile(responseTimes, 95);
      if (p95 > this.config.alertThresholds.response_time_p95) {
        alerts.push({
          type: 'THRESHOLD',
          metric: 'response_time_p95',
          value: p95,
          threshold: this.config.alertThresholds.response_time_p95,
          severity: 'MEDIUM',
          description: `P95 response time high: ${p95}ms`
        });
      }
    }
    
    return alerts;
  }
  
  checkPatternAlerts(logEntry) {
    const alerts = [];
    
    // Check for suspicious patterns
    const suspiciousPatterns = {
      repeated_failures: {
        pattern: /failed|error|exception/i,
        window: this.config.timeWindows.short,
        threshold: 20,
        severity: 'MEDIUM'
      },
      
      brute_force: {
        pattern: /(?:login\s+failed|authentication\s+failed)/i,
        window: this.config.timeWindows.medium,
        threshold: this.config.alertThresholds.failed_logins,
        severity: 'HIGH'
      },
      
      sql_injection: {
        pattern: /(?:union\s+select|drop\s+table|script\s+tag)/i,
        window: this.config.timeWindows.short,
        threshold: 1,
        severity: 'CRITICAL'
      }
    };
    
    Object.entries(suspiciousPatterns).forEach(([patternName, config]) => {
      if (config.pattern.test(logEntry.message)) {
        const recentCount = this.getPatternCountInWindow(config.pattern, config.window);
        
        if (recentCount >= config.threshold) {
          alerts.push({
            type: 'PATTERN',
            pattern: patternName,
            count: recentCount,
            threshold: config.threshold,
            severity: config.severity,
            description: `Suspicious pattern detected: ${patternName}`,
            window: config.window
          });
        }
      }
    });
    
    return alerts;
  }
  
  handleAlert(alert) {
    // Prevent duplicate alerts
    const alertKey = `${alert.type}_${alert.rule || alert.metric || alert.pattern}`;
    const lastAlert = this.alertHistory.get(alertKey);
    
    const cooldownPeriod = this.getCooldownPeriod(alert.severity);
    if (lastAlert && (Date.now() - lastAlert) < cooldownPeriod) {
      return; // Skip duplicate alert
    }
    
    this.alertHistory.set(alertKey, Date.now());
    
    // Send notifications
    this.sendNotifications(alert);
    
    // Log alert
    console.log(`[ALERT] ${alert.severity}: ${alert.description}`);
  }
  
  sendNotifications(alert) {
    this.config.notificationChannels.forEach(channel => {
      switch (channel.type) {
        case 'email':
          this.sendEmail(alert, channel.config);
          break;
        case 'slack':
          this.sendSlackMessage(alert, channel.config);
          break;
        case 'webhook':
          this.sendWebhook(alert, channel.config);
          break;
        case 'pagerduty':
          this.sendPagerDuty(alert, channel.config);
          break;
      }
    });
  }
  
  getCooldownPeriod(severity) {
    const cooldowns = {
      'CRITICAL': 2 * 60 * 1000,  // 2 minutes
      'HIGH': 5 * 60 * 1000,     // 5 minutes
      'MEDIUM': 15 * 60 * 1000,  // 15 minutes
      'LOW': 30 * 60 * 1000      // 30 minutes
    };
    
    return cooldowns[severity] || cooldowns['MEDIUM'];
  }
  
  // Utility methods
  extractMetrics(logEntry) {
    const performanceMonitor = new PerformanceMonitor();
    return performanceMonitor.extractPerformanceMetrics(logEntry.message);
  }
  
  updateMetrics(metrics) {
    const now = Date.now();
    
    Object.entries(metrics).forEach(([metricName, metricData]) => {
      if (!this.metrics.has(metricName)) {
        this.metrics.set(metricName, []);
      }
      
      this.metrics.get(metricName).push({
        value: metricData.value,
        timestamp: now
      });
      
      // Keep only recent data (last hour)
      const oneHourAgo = now - (60 * 60 * 1000);
      this.metrics.set(metricName, 
        this.metrics.get(metricName).filter(m => m.timestamp > oneHourAgo)
      );
    });
  }
  
  getMetricInWindow(metricName, windowMs) {
    const now = Date.now();
    const windowStart = now - windowMs;
    const metricData = this.metrics.get(metricName) || [];
    
    return metricData
      .filter(m => m.timestamp > windowStart)
      .map(m => m.value);
  }
  
  calculatePercentile(values, percentile) {
    const sorted = values.sort((a, b) => a - b);
    const index = Math.floor((percentile / 100) * sorted.length);
    return sorted[index] || 0;
  }
  
  getPatternCountInWindow(pattern, windowMs) {
    // This would need to be implemented based on your log storage
    // For now, returning a placeholder
    return 0;
  }
}

Performance Optimization for Log Processing

High-Performance Log Processing

class HighPerformanceLogProcessor {
  constructor(options = {}) {
    this.batchSize = options.batchSize || 1000;
    this.flushInterval = options.flushInterval || 5000; // 5 seconds
    this.maxQueueSize = options.maxQueueSize || 10000;
    
    this.logQueue = [];
    this.processing = false;
    this.stats = {
      processed: 0,
      errors: 0,
      startTime: Date.now()
    };
    
    // Pre-compile commonly used patterns
    this.compiledPatterns = this.compilePatterns();
    
    // Start periodic flush
    this.flushTimer = setInterval(() => this.flush(), this.flushInterval);
  }
  
  compilePatterns() {
    return {
      timestamp: /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?(?:Z|[+-]\d{2}:\d{2})/,
      level: /(DEBUG|INFO|WARN|WARNING|ERROR|FATAL|CRITICAL|TRACE)/i,
      ipAddress: /\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b/,
      httpStatus: /\s([1-5]\d{2})\s/,
      responseTime: /(?:response_time|duration|elapsed)[=:\s]+(\d+(?:\.\d+)?)\s*(?:ms|s)?/i,
      error: /(?:error|exception|fatal|critical)/i,
      warning: /(?:warn|warning)/i
    };
  }
  
  addLog(logEntry) {
    if (this.logQueue.length >= this.maxQueueSize) {
      console.warn('Log queue full, dropping oldest entries');
      this.logQueue = this.logQueue.slice(this.batchSize);
    }
    
    this.logQueue.push({
      ...logEntry,
      receivedAt: Date.now()
    });
    
    if (this.logQueue.length >= this.batchSize) {
      this.flush();
    }
  }
  
  async flush() {
    if (this.processing || this.logQueue.length === 0) {
      return;
    }
    
    this.processing = true;
    const batch = this.logQueue.splice(0, this.batchSize);
    
    try {
      await this.processBatch(batch);
      this.stats.processed += batch.length;
    } catch (error) {
      console.error('Error processing log batch:', error);
      this.stats.errors += batch.length;
    } finally {
      this.processing = false;
    }
  }
  
  async processBatch(logEntries) {
    // Fast path: process in parallel
    const processed = await Promise.allSettled(
      logEntries.map(entry => this.processLogEntryFast(entry))
    );
    
    // Collect results
    const results = {
      successful: [],
      failed: [],
      alerts: [],
      metrics: {}
    };
    
    processed.forEach((result, index) => {
      if (result.status === 'fulfilled') {
        const processedEntry = result.value;
        results.successful.push(processedEntry);
        
        if (processedEntry.alerts) {
          results.alerts.push(...processedEntry.alerts);
        }
        
        if (processedEntry.metrics) {
          Object.assign(results.metrics, processedEntry.metrics);
        }
      } else {
        results.failed.push({
          originalEntry: logEntries[index],
          error: result.reason.message
        });
      }
    });
    
    // Handle batch results
    await this.handleBatchResults(results);
    
    return results;
  }
  
  processLogEntryFast(logEntry) {
    const startTime = process.hrtime.bigint();
    
    try {
      const parsed = {
        timestamp: null,
        level: null,
        message: logEntry.message || logEntry.log || logEntry.text,
        metadata: {},
        alerts: [],
        metrics: {}
      };
      
      // Fast pattern matching using pre-compiled patterns
      const message = parsed.message;
      
      // Extract timestamp (try most common format first)
      const timestampMatch = message.match(this.compiledPatterns.timestamp);
      if (timestampMatch) {
        parsed.timestamp = timestampMatch[0];
      }
      
      // Extract log level
      const levelMatch = message.match(this.compiledPatterns.level);
      if (levelMatch) {
        parsed.level = levelMatch[1].toUpperCase();
      }
      
      // Extract IP address
      const ipMatch = message.match(this.compiledPatterns.ipAddress);
      if (ipMatch) {
        parsed.metadata.ip = ipMatch[0];
      }
      
      // Extract HTTP status
      const statusMatch = message.match(this.compiledPatterns.httpStatus);
      if (statusMatch) {
        parsed.metadata.status = parseInt(statusMatch[1]);
      }
      
      // Extract response time
      const responseTimeMatch = message.match(this.compiledPatterns.responseTime);
      if (responseTimeMatch) {
        parsed.metrics.responseTime = parseFloat(responseTimeMatch[1]);
      }
      
      // Fast alert checks
      if (this.compiledPatterns.error.test(message)) {
        parsed.alerts.push({
          type: 'ERROR_DETECTED',
          severity: 'HIGH',
          message: 'Error pattern detected'
        });
      }
      
      if (this.compiledPatterns.warning.test(message)) {
        parsed.alerts.push({
          type: 'WARNING_DETECTED',
          severity: 'MEDIUM',
          message: 'Warning pattern detected'
        });
      }
      
      // Performance tracking
      const endTime = process.hrtime.bigint();
      parsed.processingTime = Number(endTime - startTime) / 1000000; // Convert to milliseconds
      
      return parsed;
      
    } catch (error) {
      throw new Error(`Failed to process log entry: ${error.message}`);
    }
  }
  
  async handleBatchResults(results) {
    // Store successful entries
    if (results.successful.length > 0) {
      await this.storeLogEntries(results.successful);
    }
    
    // Handle alerts
    if (results.alerts.length > 0) {
      await this.handleAlerts(results.alerts);
    }
    
    // Update metrics
    if (Object.keys(results.metrics).length > 0) {
      await this.updateMetrics(results.metrics);
    }
    
    // Log failures
    if (results.failed.length > 0) {
      console.error(`Failed to process ${results.failed.length} log entries`);
    }
  }
  
  async storeLogEntries(entries) {
    // Implement your storage logic here
    // This could be database insertion, file writing, or forwarding to another service
    console.log(`Storing ${entries.length} log entries`);
  }
  
  async handleAlerts(alerts) {
    // Group alerts by type and severity
    const groupedAlerts = alerts.reduce((groups, alert) => {
      const key = `${alert.type}_${alert.severity}`;
      if (!groups[key]) {
        groups[key] = [];
      }
      groups[key].push(alert);
      return groups;
    }, {});
    
    // Process each group
    for (const [key, alertGroup] of Object.entries(groupedAlerts)) {
      console.log(`Processing ${alertGroup.length} alerts of type: ${key}`);
      // Implement alert handling logic
    }
  }
  
  async updateMetrics(metrics) {
    // Update performance metrics
    console.log('Updating metrics:', metrics);
  }
  
  getStats() {
    const now = Date.now();
    const uptime = now - this.stats.startTime;
    const processingRate = this.stats.processed / (uptime / 1000); // entries per second
    
    return {
      ...this.stats,
      uptime,
      processingRate: processingRate.toFixed(2),
      queueSize: this.logQueue.length,
      isProcessing: this.processing
    };
  }
  
  shutdown() {
    clearInterval(this.flushTimer);
    return this.flush(); // Final flush
  }
}

// Usage example
const processor = new HighPerformanceLogProcessor({
  batchSize: 500,
  flushInterval: 2000,
  maxQueueSize: 5000
});

// Add logs from various sources
processor.addLog({ message: '2024-01-15T10:30:45.123Z ERROR Database connection failed' });
processor.addLog({ message: '2024-01-15T10:30:46.456Z INFO Request processed successfully' });

// Get processing statistics
console.log(processor.getStats());

Real-time Log Processing

Stream Processing with Regex

class RealTimeLogProcessor {
  constructor(options = {}) {
    this.options = {
      windowSize: 60000, // 1 minute sliding window
      alertThresholds: {
        errorRate: 10,
        responseTime: 2000,
        statusCode4xx: 20,
        statusCode5xx: 5
      },
      ...options
    };
    
    this.slidingWindow = new SlidingWindow(this.options.windowSize);
    this.patternMatcher = new RealTimePatternMatcher();
    this.alertManager = new RealTimeAlertManager();
    
    this.metrics = {
      totalRequests: 0,
      errors: 0,
      responseTimes: [],
      statusCodes: {}
    };
  }
  
  processLogStream(logStream) {
    logStream.on('data', (chunk) => {
      const lines = chunk.toString().split('\n').filter(line => line.trim());
      
      lines.forEach(line => {
        this.processLogLine(line);
      });
    });
    
    logStream.on('error', (error) => {
      console.error('Log stream error:', error);
    });
    
    logStream.on('end', () => {
      console.log('Log stream ended');
    });
  }
  
  processLogLine(logLine) {
    const timestamp = Date.now();
    
    try {
      // Parse the log entry
      const parsed = this.parseLogLine(logLine);
      
      // Add to sliding window
      this.slidingWindow.add({
        timestamp,
        ...parsed
      });
      
      // Update real-time metrics
      this.updateMetrics(parsed);
      
      // Check for immediate alerts
      const alerts = this.checkRealTimeAlerts(parsed);
      alerts.forEach(alert => this.alertManager.sendAlert(alert));
      
      // Check pattern-based alerts
      const patternAlerts = this.patternMatcher.checkPatterns(logLine);
      patternAlerts.forEach(alert => this.alertManager.sendAlert(alert));
      
    } catch (error) {
      console.error('Error processing log line:', error.message);
    }
  }
  
  parseLogLine(logLine) {
    // Fast parsing with pre-compiled patterns
    const parsed = {
      level: null,
      statusCode: null,
      responseTime: null,
      ip: null,
      userAgent: null,
      method: null,
      path: null,
      isError: false
    };
    
    // Extract log level
    const levelMatch = logLine.match(/(DEBUG|INFO|WARN|ERROR|FATAL)/i);
    if (levelMatch) {
      parsed.level = levelMatch[1].toUpperCase();
      parsed.isError = ['ERROR', 'FATAL'].includes(parsed.level);
    }
    
    // Extract HTTP status code
    const statusMatch = logLine.match(/\s([1-5]\d{2})\s/);
    if (statusMatch) {
      parsed.statusCode = parseInt(statusMatch[1]);
      if (!parsed.isError && parsed.statusCode >= 400) {
        parsed.isError = true;
      }
    }
    
    // Extract response time
    const responseTimeMatch = logLine.match(/(?:response_time|duration)[=:\s]+(\d+(?:\.\d+)?)\s*(?:ms)?/i);
    if (responseTimeMatch) {
      parsed.responseTime = parseFloat(responseTimeMatch[1]);
    }
    
    // Extract IP address
    const ipMatch = logLine.match(/\b(?:\d{1,3}\.){3}\d{1,3}\b/);
    if (ipMatch) {
      parsed.ip = ipMatch[0];
    }
    
    // Extract HTTP method and path
    const httpMatch = logLine.match(/"(GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS)\s+([^\s"]+)/);
    if (httpMatch) {
      parsed.method = httpMatch[1];
      parsed.path = httpMatch[2];
    }
    
    return parsed;
  }
  
  updateMetrics(parsed) {
    this.metrics.totalRequests++;
    
    if (parsed.isError) {
      this.metrics.errors++;
    }
    
    if (parsed.responseTime) {
      this.metrics.responseTimes.push(parsed.responseTime);
      // Keep only recent response times
      if (this.metrics.responseTimes.length > 1000) {
        this.metrics.responseTimes = this.metrics.responseTimes.slice(-1000);
      }
    }
    
    if (parsed.statusCode) {
      const statusGroup = Math.floor(parsed.statusCode / 100) + 'xx';
      this.metrics.statusCodes[statusGroup] = (this.metrics.statusCodes[statusGroup] || 0) + 1;
    }
  }
  
  checkRealTimeAlerts(parsed) {
    const alerts = [];
    const windowData = this.slidingWindow.getData();
    
    // Check error rate
    const errors = windowData.filter(entry => entry.isError).length;
    const errorRate = errors / Math.max(1, windowData.length) * 100;
    
    if (errorRate > this.options.alertThresholds.errorRate) {
      alerts.push({
        type: 'HIGH_ERROR_RATE',
        severity: 'HIGH',
        value: errorRate,
        threshold: this.options.alertThresholds.errorRate,
        description: `Error rate: ${errorRate.toFixed(2)}%`
      });
    }
    
    // Check response time
    if (parsed.responseTime && parsed.responseTime > this.options.alertThresholds.responseTime) {
      alerts.push({
        type: 'HIGH_RESPONSE_TIME',
        severity: 'MEDIUM',
        value: parsed.responseTime,
        threshold: this.options.alertThresholds.responseTime,
        description: `High response time: ${parsed.responseTime}ms`
      });
    }
    
    // Check 4xx status codes
    if (parsed.statusCode >= 400 && parsed.statusCode < 500) {
      const count4xx = windowData.filter(entry => 
        entry.statusCode >= 400 && entry.statusCode < 500
      ).length;
      
      if (count4xx > this.options.alertThresholds.statusCode4xx) {
        alerts.push({
          type: 'HIGH_4XX_RATE',
          severity: 'MEDIUM',
          value: count4xx,
          threshold: this.options.alertThresholds.statusCode4xx,
          description: `High 4xx error count: ${count4xx}`
        });
      }
    }
    
    // Check 5xx status codes
    if (parsed.statusCode >= 500) {
      const count5xx = windowData.filter(entry => entry.statusCode >= 500).length;
      
      if (count5xx > this.options.alertThresholds.statusCode5xx) {
        alerts.push({
          type: 'HIGH_5XX_RATE',
          severity: 'CRITICAL',
          value: count5xx,
          threshold: this.options.alertThresholds.statusCode5xx,
          description: `High 5xx error count: ${count5xx}`
        });
      }
    }
    
    return alerts;
  }
  
  getRealTimeMetrics() {
    const windowData = this.slidingWindow.getData();
    const now = Date.now();
    const oneMinuteAgo = now - 60000;
    
    const recentData = windowData.filter(entry => entry.timestamp > oneMinuteAgo);
    
    const responseTimes = recentData
      .filter(entry => entry.responseTime)
      .map(entry => entry.responseTime);
    
    return {
      timestamp: now,
      windowSize: recentData.length,
      totalRequests: this.metrics.totalRequests,
      errorRate: recentData.filter(entry => entry.isError).length / Math.max(1, recentData.length) * 100,
      avgResponseTime: responseTimes.length > 0 ? 
        responseTimes.reduce((a, b) => a + b) / responseTimes.length : 0,
      p95ResponseTime: this.calculatePercentile(responseTimes, 95),
      statusCodeDistribution: this.getStatusCodeDistribution(recentData)
    };
  }
  
  calculatePercentile(values, percentile) {
    if (values.length === 0) return 0;
    const sorted = values.sort((a, b) => a - b);
    const index = Math.floor((percentile / 100) * sorted.length);
    return sorted[index] || 0;
  }
  
  getStatusCodeDistribution(data) {
    const distribution = {};
    data.forEach(entry => {
      if (entry.statusCode) {
        const group = Math.floor(entry.statusCode / 100) + 'xx';
        distribution[group] = (distribution[group] || 0) + 1;
      }
    });
    return distribution;
  }
}

// Supporting classes
class SlidingWindow {
  constructor(windowSize) {
    this.windowSize = windowSize;
    this.data = [];
  }
  
  add(entry) {
    this.data.push(entry);
    this.cleanup();
  }
  
  cleanup() {
    const now = Date.now();
    const cutoff = now - this.windowSize;
    this.data = this.data.filter(entry => entry.timestamp > cutoff);
  }
  
  getData() {
    this.cleanup();
    return this.data;
  }
}

class RealTimePatternMatcher {
  constructor() {
    this.patterns = {
      sqlInjection: /(?:union\s+select|drop\s+table|delete\s+from)/i,
      xssAttack: /(?: {
      if (pattern.test(logLine)) {
        this.patternCounts[patternName] = (this.patternCounts[patternName] || 0) + 1;
        
        alerts.push({
          type: 'PATTERN_MATCH',
          pattern: patternName,
          severity: this.getPatternSeverity(patternName),
          description: `Suspicious pattern detected: ${patternName}`,
          count: this.patternCounts[patternName]
        });
      }
    });
    
    return alerts;
  }
  
  getPatternSeverity(patternName) {
    const severities = {
      sqlInjection: 'CRITICAL',
      xssAttack: 'HIGH',
      bruteForce: 'HIGH',
      suspiciousUserAgent: 'LOW'
    };
    
    return severities[patternName] || 'MEDIUM';
  }
}

class RealTimeAlertManager {
  constructor() {
    this.recentAlerts = new Map();
    this.cooldownPeriod = 60000; // 1 minute
  }
  
  sendAlert(alert) {
    const alertKey = `${alert.type}_${alert.pattern || alert.value}`;
    const lastAlert = this.recentAlerts.get(alertKey);
    
    if (lastAlert && (Date.now() - lastAlert) < this.cooldownPeriod) {
      return; // Skip duplicate alert
    }
    
    this.recentAlerts.set(alertKey, Date.now());
    
    // Send the alert (implement your notification logic here)
    console.log(`[REALTIME ALERT] ${alert.severity}: ${alert.description}`);
    
    // Clean up old alerts
    this.cleanupOldAlerts();
  }
  
  cleanupOldAlerts() {
    const now = Date.now();
    const cutoff = now - this.cooldownPeriod;
    
    for (const [key, timestamp] of this.recentAlerts.entries()) {
      if (timestamp < cutoff) {
        this.recentAlerts.delete(key);
      }
    }
  }
}

Security Log Analysis

Security Event Detection

class SecurityLogAnalyzer {
  constructor() {
    this.securityPatterns = {
      // Authentication and Authorization
      authFailure: /(?:authentication|login|auth).*?(?:failed|denied|rejected|invalid)/i,
      authSuccess: /(?:authentication|login|auth).*?(?:successful|succeeded|accepted|valid)/i,
      privilegeEscalation: /(?:sudo|su|elevation|privilege).*?(?:executed|granted|escalated)/i,
      
      // Attack Patterns
      sqlInjection: /(?:union\s+select|drop\s+table|insert\s+into|delete\s+from|--|'\s*or\s*')/i,
      xssAttempt: /(?: {
      const matches = logText.match(pattern);
      if (matches) {
        const severity = this.getSecuritySeverity(patternName);
        const event = {
          type: 'SECURITY_EVENT',
          pattern: patternName,
          severity,
          timestamp,
          ip,
          userAgent,
          details: matches[0],
          logEntry: logText.substring(0, 200) + (logText.length > 200 ? '...' : ''),
          category: this.categorizeSecurityEvent(patternName)
        };
        
        securityEvents.push(event);
        
        // Update threat tracking
        if (ip) {
          this.updateIPThreatLevel(ip, patternName, severity);
        }
        
        // Track attack patterns
        this.trackAttackPattern(patternName, ip);
      }
    });
    
    // Analyze attack correlations
    const correlatedAttacks = this.analyzeAttackCorrelations(ip, securityEvents);
    
    return {
      events: securityEvents,
      correlatedAttacks,
      ipThreatLevel: ip ? this.getIPThreatLevel(ip) : null
    };
  }
  
  getSecuritySeverity(patternName) {
    for (const [severity, patterns] of Object.entries(this.severityLevels)) {
      if (patterns.includes(patternName)) {
        return severity;
      }
    }
    return 'UNKNOWN';
  }
  
  categorizeSecurityEvent(patternName) {
    const categories = {
      'Authentication': ['authFailure', 'authSuccess', 'privilegeEscalation', 'bruteForceSSH', 'bruteForceWeb', 'bruteForceDB'],
      'Injection': ['sqlInjection', 'xssAttempt', 'commandInjection'],
      'Access Control': ['unauthorizedAccess', 'pathTraversal', 'privilegedAccess'],
      'Malware': ['malwareSignature', 'suspiciousFile'],
      'Network': ['portScan', 'ddosAttempt', 'suspiciousTraffic'],
      'Data Protection': ['dataExfiltration'],
      'Behavioral': ['offHoursActivity']
    };
    
    for (const [category, patterns] of Object.entries(categories)) {
      if (patterns.includes(patternName)) {
        return category;
      }
    }
    
    return 'General';
  }
  
  updateIPThreatLevel(ip, patternName, severity) {
    if (!this.ipThreatMap.has(ip)) {
      this.ipThreatMap.set(ip, {
        threatScore: 0,
        events: [],
        firstSeen: Date.now(),
        lastSeen: Date.now()
      });
    }
    
    const ipData = this.ipThreatMap.get(ip);
    
    // Update threat score based on severity
    const scoreMap = {
      'CRITICAL': 10,
      'HIGH': 5,
      'MEDIUM': 2,
      'LOW': 1
    };
    
    ipData.threatScore += scoreMap[severity] || 1;
    ipData.events.push({ pattern: patternName, severity, timestamp: Date.now() });
    ipData.lastSeen = Date.now();
    
    // Keep only recent events (last 24 hours)
    const oneDayAgo = Date.now() - (24 * 60 * 60 * 1000);
    ipData.events = ipData.events.filter(event => event.timestamp > oneDayAgo);
    
    this.ipThreatMap.set(ip, ipData);
  }
  
  getIPThreatLevel(ip) {
    const ipData = this.ipThreatMap.get(ip);
    if (!ipData) return 'UNKNOWN';
    
    const score = ipData.threatScore;
    
    if (score >= 50) return 'CRITICAL';
    if (score >= 20) return 'HIGH';
    if (score >= 10) return 'MEDIUM';
    if (score >= 5) return 'LOW';
    
    return 'MINIMAL';
  }
  
  trackAttackPattern(patternName, ip) {
    const key = `${patternName}_${ip || 'unknown'}`;
    if (!this.attackCounters[key]) {
      this.attackCounters[key] = {
        count: 0,
        firstSeen: Date.now(),
        lastSeen: Date.now()
      };
    }
    
    this.attackCounters[key].count++;
    this.attackCounters[key].lastSeen = Date.now();
  }
  
  analyzeAttackCorrelations(ip, currentEvents) {
    const correlations = [];
    
    if (!ip || currentEvents.length === 0) {
      return correlations;
    }
    
    // Check for coordinated attacks
    const ipData = this.ipThreatMap.get(ip);
    if (ipData && ipData.events.length > 5) {
      const recentEvents = ipData.events.filter(event => 
        Date.now() - event.timestamp < (5 * 60 * 1000) // Last 5 minutes
      );
      
      if (recentEvents.length > 3) {
        correlations.push({
          type: 'COORDINATED_ATTACK',
          severity: 'CRITICAL',
          description: `Multiple attack patterns from IP ${ip} in short timeframe`,
          eventCount: recentEvents.length,
          patterns: [...new Set(recentEvents.map(e => e.pattern))]
        });
      }
    }
    
    // Check for escalation patterns
    const currentPatterns = currentEvents.map(e => e.pattern);
    if (currentPatterns.includes('authFailure') && currentPatterns.includes('privilegeEscalation')) {
      correlations.push({
        type: 'PRIVILEGE_ESCALATION_SEQUENCE',
        severity: 'HIGH',
        description: 'Authentication failure followed by privilege escalation attempt',
        patterns: ['authFailure', 'privilegeEscalation']
      });
    }
    
    // Check for data exfiltration after unauthorized access
    if (currentPatterns.includes('unauthorizedAccess') && currentPatterns.includes('dataExfiltration')) {
      correlations.push({
        type: 'DATA_BREACH_SEQUENCE',
        severity: 'CRITICAL',
        description: 'Unauthorized access followed by potential data exfiltration',
        patterns: ['unauthorizedAccess', 'dataExfiltration']
      });
    }
    
    return correlations;
  }
  
  generateSecurityReport(timeRange = 24 * 60 * 60 * 1000) { // 24 hours default
    const now = Date.now();
    const startTime = now - timeRange;
    
    const report = {
      timestamp: new Date(now).toISOString(),
      timeRange: `${timeRange / (60 * 60 * 1000)} hours`,
      summary: {
        totalEvents: 0,
        severityBreakdown: { CRITICAL: 0, HIGH: 0, MEDIUM: 0, LOW: 0 },
        topAttackTypes: {},
        threatIPs: []
      },
      details: {
        attackPatterns: {},
        ipAnalysis: {},
        correlatedAttacks: []
      }
    };
    
    // Analyze IP threat data
    for (const [ip, ipData] of this.ipThreatMap.entries()) {
      const recentEvents = ipData.events.filter(event => event.timestamp > startTime);
      
      if (recentEvents.length > 0) {
        report.summary.totalEvents += recentEvents.length;
        
        // Update severity breakdown
        recentEvents.forEach(event => {
          report.summary.severityBreakdown[event.severity]++;
        });
        
        // Track attack types
        recentEvents.forEach(event => {
          report.summary.topAttackTypes[event.pattern] = 
            (report.summary.topAttackTypes[event.pattern] || 0) + 1;
        });
        
        // Add to threat IPs if significant activity
        if (recentEvents.length > 3 || ipData.threatScore > 10) {
          report.summary.threatIPs.push({
            ip,
            threatLevel: this.getIPThreatLevel(ip),
            eventCount: recentEvents.length,
            threatScore: ipData.threatScore
          });
        }
        
        report.details.ipAnalysis[ip] = {
          threatLevel: this.getIPThreatLevel(ip),
          threatScore: ipData.threatScore,
          events: recentEvents,
          firstSeen: new Date(ipData.firstSeen).toISOString(),
          lastSeen: new Date(ipData.lastSeen).toISOString()
        };
      }
    }
    
    // Sort and limit top results
    report.summary.threatIPs = report.summary.threatIPs
      .sort((a, b) => b.threatScore - a.threatScore)
      .slice(0, 10);
    
    return report;
  }
  
  extractIP(logText) {
    const ipPattern = /\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b/;
    const match = logText.match(ipPattern);
    return match ? match[0] : null;
  }
  
  extractTimestamp(logText) {
    const timestampPattern = /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?(?:Z|[+-]\d{2}:\d{2})/;
    const match = logText.match(timestampPattern);
    return match ? match[0] : new Date().toISOString();
  }
  
  extractUserAgent(logText) {
    const userAgentPattern = /"([^"]*(?:Mozilla|Chrome|Safari|Firefox|Edge|Opera|Bot|Spider|Crawler)[^"]*)"/i;
    const match = logText.match(userAgentPattern);
    return match ? match[1] : null;
  }
}

Troubleshooting and Debug Patterns

Debugging Assistant

class LogTroubleshootingAssistant {
  constructor() {
    this.troubleshootingPatterns = {
      // Connection Issues
      connectionTimeout: {
        pattern: /connection.*?(?:timeout|timed\s+out|failed)/i,
        category: 'Connectivity',
        suggestions: [
          'Check network connectivity between services',
          'Verify firewall rules and port accessibility',
          'Increase connection timeout values',
          'Check service health and availability'
        ]
      },
      
      connectionRefused: {
        pattern: /connection.*?refused/i,
        category: 'Connectivity',
        suggestions: [
          'Verify target service is running',
          'Check if service is listening on correct port',
          'Verify network routing and DNS resolution',
          'Check security group/firewall configurations'
        ]
      },
      
      // Database Issues
      databaseConnectionFailed: {
        pattern: /(?:database|db).*?connection.*?(?:failed|error|lost)/i,
        category: 'Database',
        suggestions: [
          'Check database server status and logs',
          'Verify connection string and credentials',
          'Check connection pool settings',
          'Monitor database resource usage (CPU, memory, disk)'
        ]
      },
      
      sqlSyntaxError: {
        pattern: /sql.*?syntax.*?error/i,
        category: 'Database',
        suggestions: [
          'Review SQL query syntax',
          'Check for SQL injection attempts',
          'Validate database schema compatibility',
          'Check for reserved keywords usage'
        ]
      },
      
      deadlockDetected: {
        pattern: /deadlock|lock\s+timeout/i,
        category: 'Database',
        suggestions: [
          'Review transaction isolation levels',
          'Optimize query execution order',
          'Consider implementing retry logic',
          'Monitor database locks and wait events'
        ]
      },
      
      // Memory Issues
      outOfMemory: {
        pattern: /(?:out\s+of\s+memory|OutOfMemoryError|heap\s+space)/i,
        category: 'Memory',
        suggestions: [
          'Increase JVM heap size or container memory limits',
          'Analyze memory usage patterns and potential leaks',
          'Review garbage collection settings',
          'Optimize data structures and caching strategies'
        ]
      },
      
      memoryLeak: {
        pattern: /memory\s+leak/i,
        category: 'Memory',
        suggestions: [
          'Use memory profiling tools to identify leak sources',
          'Review object lifecycle management',
          'Check for unclosed resources (files, connections)',
          'Monitor memory usage trends over time'
        ]
      },
      
      // Performance Issues
      slowQuery: {
        pattern: /(?:slow\s+query|query.*?exceeded|query.*?timeout)/i,
        category: 'Performance',
        suggestions: [
          'Analyze query execution plan',
          'Add appropriate database indexes',
          'Consider query optimization or rewriting',
          'Monitor database resource utilization'
        ]
      },
      
      highLatency: {
        pattern: /(?:high\s+latency|response\s+time.*?high|latency.*?exceeded)/i,
        category: 'Performance',
        suggestions: [
          'Profile application performance bottlenecks',
          'Check network latency between services',
          'Review caching strategies',
          'Consider load balancing and scaling options'
        ]
      },
      
      // Security Issues
      authenticationFailed: {
        pattern: /authentication.*?(?:failed|denied|invalid)/i,
        category: 'Security',
        suggestions: [
          'Verify credentials and authentication configuration',
          'Check token expiration and renewal logic',
          'Review authentication service logs',
          'Monitor for potential brute force attacks'
        ]
      },
      
      certificateError: {
        pattern: /(?:certificate|ssl|tls).*?(?:error|expired|invalid)/i,
        category: 'Security',
        suggestions: [
          'Check certificate validity and expiration dates',
          'Verify certificate chain and trust store',
          'Review SSL/TLS configuration',
          'Check certificate authority and domain matching'
        ]
      },
      
      // Configuration Issues
      configurationError: {
        pattern: /(?:configuration|config).*?(?:error|invalid|missing)/i,
        category: 'Configuration',
        suggestions: [
          'Validate configuration file syntax and structure',
          'Check for missing required configuration values',
          'Verify environment variable settings',
          'Review configuration inheritance and overrides'
        ]
      },
      
      permissionDenied: {
        pattern: /permission.*?denied/i,
        category: 'Configuration',
        suggestions: [
          'Check file and directory permissions',
          'Verify user account privileges',
          'Review security policies and access controls',
          'Check service account permissions'
        ]
      }
    };
  }
  
  analyzeTroubleshootingOpportunities(logEntries) {
    const issues = [];
    const issueCounts = {};
    
    logEntries.forEach(entry => {
      const logText = entry.message || entry.log || String(entry);
      
      Object.entries(this.troubleshootingPatterns).forEach(([issueName, issueConfig]) => {
        if (issueConfig.pattern.test(logText)) {
          const issue = {
            type: issueName,
            category: issueConfig.category,
            timestamp: entry.timestamp || new Date().toISOString(),
            logEntry: logText,
            suggestions: issueConfig.suggestions,
            severity: this.determineSeverity(issueName),
            extractedInfo: this.extractRelevantInfo(logText, issueName)
          };
          
          issues.push(issue);
          issueCounts[issueName] = (issueCounts[issueName] || 0) + 1;
        }
      });
    });
    
    return {
      issues,
      summary: this.generateTroubleshootingSummary(issues, issueCounts),
      recommendations: this.generateRecommendations(issues)
    };
  }
  
  determineSeverity(issueName) {
    const severityMap = {
      outOfMemory: 'CRITICAL',
      databaseConnectionFailed: 'CRITICAL',
      deadlockDetected: 'HIGH',
      connectionRefused: 'HIGH',
      sqlSyntaxError: 'HIGH',
      authenticationFailed: 'HIGH',
      certificateError: 'HIGH',
      connectionTimeout: 'MEDIUM',
      slowQuery: 'MEDIUM',
      highLatency: 'MEDIUM',
      memoryLeak: 'MEDIUM',
      configurationError: 'MEDIUM',
      permissionDenied: 'MEDIUM'
    };
    
    return severityMap[issueName] || 'LOW';
  }
  
  extractRelevantInfo(logText, issueName) {
    const extractors = {
      connectionTimeout: () => {
        const timeoutMatch = logText.match(/(\d+)\s*(?:ms|milliseconds|s|seconds)/i);
        const hostMatch = logText.match(/(?:host|server|endpoint)\s*:?\s*([\w.-]+)/i);
        return {
          timeout: timeoutMatch ? timeoutMatch[1] : null,
          host: hostMatch ? hostMatch[1] : null
        };
      },
      
      databaseConnectionFailed: () => {
        const dbMatch = logText.match(/(?:database|db)\s*:?\s*([\w.-]+)/i);
        const portMatch = logText.match(/port\s*:?\s*(\d+)/i);
        return {
          database: dbMatch ? dbMatch[1] : null,
          port: portMatch ? portMatch[1] : null
        };
      },
      
      slowQuery: () => {
        const timeMatch = logText.match(/(\d+(?:\.\d+)?)\s*(?:ms|s|seconds)/i);
        const queryMatch = logText.match(/(?:query|sql)\s*:?\s*([^\n]{1,100})/i);
        return {
          executionTime: timeMatch ? parseFloat(timeMatch[1]) : null,
          query: queryMatch ? queryMatch[1].trim() : null
        };
      },
      
      authenticationFailed: () => {
        const userMatch = logText.match(/(?:user|username)\s*:?\s*([\w@.-]+)/i);
        const ipMatch = logText.match(/\b(?:\d{1,3}\.){3}\d{1,3}\b/);
        return {
          user: userMatch ? userMatch[1] : null,
          ip: ipMatch ? ipMatch[0] : null
        };
      }
    };
    
    const extractor = extractors[issueName];
    return extractor ? extractor() : {};
  }
  
  generateTroubleshootingSummary(issues, issueCounts) {
    const categoryBreakdown = {};
    const severityBreakdown = { CRITICAL: 0, HIGH: 0, MEDIUM: 0, LOW: 0 };
    
    issues.forEach(issue => {
      categoryBreakdown[issue.category] = (categoryBreakdown[issue.category] || 0) + 1;
      severityBreakdown[issue.severity]++;
    });
    
    const topIssues = Object.entries(issueCounts)
      .sort((a, b) => b[1] - a[1])
      .slice(0, 5)
      .map(([issue, count]) => ({ issue, count }));
    
    return {
      totalIssues: issues.length,
      uniqueIssueTypes: Object.keys(issueCounts).length,
      categoryBreakdown,
      severityBreakdown,
      topIssues
    };
  }
  
  generateRecommendations(issues) {
    const recommendations = [];
    const categoryIssues = {};
    
    // Group issues by category
    issues.forEach(issue => {
      if (!categoryIssues[issue.category]) {
        categoryIssues[issue.category] = [];
      }
      categoryIssues[issue.category].push(issue);
    });
    
    // Generate category-specific recommendations
    Object.entries(categoryIssues).forEach(([category, categoryIssuesArray]) => {
      const criticalIssues = categoryIssuesArray.filter(issue => issue.severity === 'CRITICAL');
      const highIssues = categoryIssuesArray.filter(issue => issue.severity === 'HIGH');
      
      if (criticalIssues.length > 0) {
        recommendations.push({
          priority: 'CRITICAL',
          category,
          title: `Address Critical ${category} Issues`,
          description: `${criticalIssues.length} critical ${category.toLowerCase()} issues require immediate attention`,
          actions: this.getImmediateActions(criticalIssues)
        });
      }
      
      if (highIssues.length > 0) {
        recommendations.push({
          priority: 'HIGH',
          category,
          title: `Resolve ${category} Problems`,
          description: `${highIssues.length} high-priority ${category.toLowerCase()} issues detected`,
          actions: this.getRecommendedActions(highIssues)
        });
      }
      
      // Look for patterns that suggest systemic issues
      if (categoryIssuesArray.length > 10) {
        recommendations.push({
          priority: 'MEDIUM',
          category,
          title: `Review ${category} Architecture`,
          description: `High frequency of ${category.toLowerCase()} issues suggests potential systemic problems`,
          actions: this.getSystemicRecommendations(category)
        });
      }
    });
    
    return recommendations.sort((a, b) => {
      const priorityOrder = { CRITICAL: 0, HIGH: 1, MEDIUM: 2, LOW: 3 };
      return priorityOrder[a.priority] - priorityOrder[b.priority];
    });
  }
  
  getImmediateActions(issues) {
    const actions = new Set();
    
    issues.forEach(issue => {
      switch (issue.type) {
        case 'outOfMemory':
          actions.add('Immediately increase memory allocation');
          actions.add('Restart affected services');
          break;
        case 'databaseConnectionFailed':
          actions.add('Check database server health');
          actions.add('Verify database connectivity');
          break;
        case 'deadlockDetected':
          actions.add('Monitor active database sessions');
          actions.add('Consider implementing circuit breaker pattern');
          break;
      }
    });
    
    return Array.from(actions);
  }
  
  getRecommendedActions(issues) {
    const actions = new Set();
    
    issues.forEach(issue => {
      issue.suggestions.slice(0, 2).forEach(suggestion => {
        actions.add(suggestion);
      });
    });
    
    return Array.from(actions);
  }
  
  getSystemicRecommendations(category) {
    const recommendations = {
      'Connectivity': [
        'Implement comprehensive network monitoring',
        'Review service mesh or load balancer configuration',
        'Consider implementing retry and circuit breaker patterns'
      ],
      'Database': [
        'Conduct database performance review',
        'Implement database monitoring and alerting',
        'Review connection pooling and timeout configurations'
      ],
      'Memory': [
        'Implement memory monitoring and profiling',
        'Review garbage collection tuning',
        'Consider horizontal scaling strategies'
      ],
      'Performance': [
        'Conduct comprehensive performance testing',
        'Implement application performance monitoring (APM)',
        'Review caching strategies and optimization opportunities'
      ],
      'Security': [
        'Review security configurations and policies',
        'Implement comprehensive security monitoring',
        'Conduct security audit and penetration testing'
      ],
      'Configuration': [
        'Implement configuration management and validation',
        'Review deployment and configuration processes',
        'Consider infrastructure as code approaches'
      ]
    };
    
    return recommendations[category] || ['Review system architecture and design patterns'];
  }
  
  generateDiagnosticReport(logEntries) {
    const analysis = this.analyzeTroubleshootingOpportunities(logEntries);
    
    const report = {
      timestamp: new Date().toISOString(),
      summary: analysis.summary,
      prioritizedIssues: this.prioritizeIssues(analysis.issues),
      recommendations: analysis.recommendations,
      diagnosticSteps: this.generateDiagnosticSteps(analysis.issues),
      preventiveMeasures: this.generatePreventiveMeasures(analysis.issues)
    };
    
    return report;
  }
  
  prioritizeIssues(issues) {
    return issues
      .sort((a, b) => {
        const priorityOrder = { CRITICAL: 0, HIGH: 1, MEDIUM: 2, LOW: 3 };
        return priorityOrder[a.severity] - priorityOrder[b.severity];
      })
      .slice(0, 20); // Top 20 issues
  }
  
  generateDiagnosticSteps(issues) {
    const steps = [];
    const criticalIssues = issues.filter(issue => issue.severity === 'CRITICAL');
    
    if (criticalIssues.length > 0) {
      steps.push({
        step: 1,
        title: 'Address Critical Issues',
        description: 'Immediately investigate and resolve critical system issues',
        tasks: criticalIssues.map(issue => `Resolve ${issue.type}: ${issue.suggestions[0]}`)
      });
    }
    
    const connectivityIssues = issues.filter(issue => issue.category === 'Connectivity');
    if (connectivityIssues.length > 0) {
      steps.push({
        step: steps.length + 1,
        title: 'Verify Network Connectivity',
        description: 'Test network connectivity between services and external dependencies',
        tasks: [
          'Run network connectivity tests',
          'Check DNS resolution',
          'Verify firewall and security group settings',
          'Test service endpoints'
        ]
      });
    }
    
    return steps;
  }
  
  generatePreventiveMeasures(issues) {
    const measures = [];
    const categories = [...new Set(issues.map(issue => issue.category))];
    
    categories.forEach(category => {
      switch (category) {
        case 'Database':
          measures.push({
            category,
            measures: [
              'Implement database connection pooling',
              'Set up database monitoring and alerting',
              'Regular database performance tuning',
              'Implement automated backup and recovery testing'
            ]
          });
          break;
        case 'Memory':
          measures.push({
            category,
            measures: [
              'Implement memory usage monitoring',
              'Set up automated garbage collection tuning',
              'Regular memory leak detection',
              'Implement memory-efficient coding practices'
            ]
          });
          break;
        case 'Performance':
          measures.push({
            category,
            measures: [
              'Implement performance testing in CI/CD',
              'Set up application performance monitoring',
              'Regular performance baseline reviews',
              'Implement caching strategies'
            ]
          });
          break;
      }
    });
    
    return measures;
  }
}

// Usage example
const troubleshooter = new LogTroubleshootingAssistant();

const sampleLogs = [
  { message: '2024-01-15T10:30:45Z ERROR Database connection failed: timeout after 30s' },
  { message: '2024-01-15T10:31:00Z ERROR OutOfMemoryError: Java heap space' },
  { message: '2024-01-15T10:31:15Z WARN Slow query detected: SELECT * FROM users took 2.5s' },
  { message: '2024-01-15T10:31:30Z ERROR Authentication failed for user john@example.com from 192.168.1.100' }
];

const diagnosticReport = troubleshooter.generateDiagnosticReport(sampleLogs);
console.log('Diagnostic Report:', JSON.stringify(diagnosticReport, null, 2));

Conclusion

Mastering regex for log parsing and monitoring requires understanding both the technical patterns and the operational context. Key takeaways:

🎯 Best Practices for Log Regex:

• Performance First: Pre-compile patterns and use efficient matching strategies
• Pattern Hierarchy: Start with fast, simple patterns before complex ones
• Context Awareness: Understand your log formats and adapt patterns accordingly
• Error Handling: Always handle regex failures gracefully

📊 Monitoring Strategies:

• Real-time Processing: Use sliding windows and streaming approaches
• Alert Correlation: Connect related events for better insights
• Performance Metrics: Track processing speed and resource usage
• Security Focus: Prioritize security events and threat detection

🔧 Troubleshooting Approach:

• Pattern Recognition: Use regex to identify known issue signatures
• Context Extraction: Pull relevant details for diagnosis
• Severity Assessment: Prioritize issues based on business impact
• Actionable Insights: Provide specific recommendations

⚡ Performance Optimization:

• Batch Processing: Process logs in batches for efficiency
• Caching: Cache parsed results and pattern compilations
• Parallel Processing: Use multiple workers for high-volume logs
• Resource Management: Monitor memory and CPU usage

Effective log analysis with regex enables proactive system monitoring, rapid issue resolution, and continuous improvement of system reliability. Regular review and optimization of your regex patterns ensures they remain effective as your systems evolve.