Energy-Efficient Auto-Scaling Mechanisms for Big Data Workloads in Cloud Environments: A Case Study of Apache Spark on Kubernetes

Abstract

Energy consumption has become a critical concern in cloud computing platforms that handle large-scale data processing workloads. This work explores the design of energy-efficient auto-scaling mechanisms for big data workloads in cloud environments, with a focus on optimizing resource allocation for Apache Spark running on Kubernetes clusters. The approach involves formulating a modeling framework that captures dynamic workload behavior, resource usage patterns, and node energy consumption characteristics. By continuously monitoring workload intensity, the mechanism predicts future fluctuations in computational demand, adjusts the number of allocated worker nodes, and orchestrates container-based resources to minimize energy overhead while meeting stringent performance constraints. The framework is validated through extensive simulations and empirical testing in Kubernetes-based infrastructures, highlighting the trade-offs between runtime efficiency, throughput, and power utilization across varying workload profiles. Results suggest that a well-structured auto-scaling strategy offers a substantial reduction in energy consumption without sacrificing crucial performance requirements such as latency or completion time. The analysis sheds light on tuning parameters, such as scaling thresholds and resource utilization baselines, that significantly affect the energy-performance trade-off. The findings emphasize the central role of data-driven predictive models in shaping cluster provisioning strategies. The study demonstrates the potential of advanced scaling algorithms to empower sustainable, energy-aware big data processing in modern cloud architectures.