Browsing by Author "Shabestari, Fatemeh"

Now showing 1 - 2 of 2

Citation Count: 0
An Energy-Aware Resource Management Strategy Based on Spark and YARN in Heterogeneous Environments
(Ieee-inst Electrical Electronics Engineers inc, 2024) Shabestari, Fatemeh; Navimipour, Nima Jafari
Apache Spark is a popular framework for processing big data. Running Spark on Hadoop YARN allows it to schedule Spark workloads alongside other data-processing frameworks on Hadoop. When an application is deployed in a YARN cluster, its resources are given without considering energy efficiency. Furthermore, there is no way to enforce any user-specified deadline constraints. To address these issues, we propose a new deadline-aware resource management system and a scheduling algorithm to minimize the total energy consumption in Spark on YARN for heterogeneous clusters. First, a deadline-aware energy-efficient model for the considered problem is proposed. Then, using a locality-aware method, executors are assigned to applications. This algorithm sorts the nodes based on the performance per watt (PPW) metric, the number of application data blocks on nodes, and the rack locality. It also offers three ways to choose executors from different machines: greedy, random, and Pareto-based. Finally, the proposed heuristic task scheduler schedules tasks on executors to minimize total energy and tardiness. We evaluated the performance of the suggested algorithm regarding energy efficiency and satisfying the Service Level Agreement (SLA). The results showed that the method outperforms the popular algorithms regarding energy consumption and meeting deadlines.
Citation Count: 3
A Yarn-Based Energy-Aware Scheduling Method for Big Data Applications Under Deadline Constraints
(Springer, 2022) Shabestari, Fatemeh; Rahmani, Amir Masoud; Navimipour, Nima Jafari; Jabbehdari, Sam
Hadoop is a distributed framework for processing big data. One of the critical parts of Hadoop is YARN, which carries out scheduling and resource management. A scheduling algorithm should consider multiple objectives. However, YARN schedulers do not consider the Service Level Agreement (SLA) and the energy-related issues. The present paper proposes an energy-efficient deadline-aware model for the scheduling problem. The scheduling issue is an NP-hard problem regarding the deadline of applications and reducing energy. Hence, an Energy-efficient Deadline-aware Scheduling Algorithm based on the Moth-Flame Optimization algorithm (EDSA-MFO) is suggested to minimize the energy consumption and execute the application within a given soft deadline. Moreover, the earliest deadline first-based (EDF-based) heuristic approach is proposed to decode a moth into a scheduling solution. The algorithm is implemented for both static and dynamic scheduling. To evaluate the performance of the proposed algorithm, extensive simulations are conducted. The outcomes demonstrated that the suggested method could find near-optimal scheduling. It outperforms the YARN default FIFO scheduler, EDF, the energy-aware greedy algorithm (EAGA), and the deadline-aware energy-efficient MapReduce scheduling algorithm for YARN (EMRSAY) in total cluster energy consumption and meeting job deadline.