An Energy-Aware Resource Management Strategy Based on Spark and YARN in Heterogeneous Environments

dc.authoridShabestari, Fatemeh/0000-0003-1926-4674
dc.authorscopusid57204862467
dc.authorscopusid55897274300
dc.contributor.authorShabestari, Fatemeh
dc.contributor.authorNavimipour, Nima Jafari
dc.date.accessioned2024-06-23T21:38:08Z
dc.date.available2024-06-23T21:38:08Z
dc.date.issued2024
dc.departmentKadir Has Universityen_US
dc.department-temp[Shabestari, Fatemeh] Islamic Azad Univ, Dept Comp Engn, Sofian Branch, Sofian, Iran; [Navimipour, Nima Jafari] Kadir Has Univ, Dept Comp Engn, TR-34083 Istanbul, Turkiye; [Navimipour, Nima Jafari] Natl Yunlin Univ Sci & Technol, Future Technol Res Ctr, Touliu 64002, Taiwanen_US
dc.descriptionShabestari, Fatemeh/0000-0003-1926-4674en_US
dc.description.abstractApache Spark is a popular framework for processing big data. Running Spark on Hadoop YARN allows it to schedule Spark workloads alongside other data-processing frameworks on Hadoop. When an application is deployed in a YARN cluster, its resources are given without considering energy efficiency. Furthermore, there is no way to enforce any user-specified deadline constraints. To address these issues, we propose a new deadline-aware resource management system and a scheduling algorithm to minimize the total energy consumption in Spark on YARN for heterogeneous clusters. First, a deadline-aware energy-efficient model for the considered problem is proposed. Then, using a locality-aware method, executors are assigned to applications. This algorithm sorts the nodes based on the performance per watt (PPW) metric, the number of application data blocks on nodes, and the rack locality. It also offers three ways to choose executors from different machines: greedy, random, and Pareto-based. Finally, the proposed heuristic task scheduler schedules tasks on executors to minimize total energy and tardiness. We evaluated the performance of the suggested algorithm regarding energy efficiency and satisfying the Service Level Agreement (SLA). The results showed that the method outperforms the popular algorithms regarding energy consumption and meeting deadlines.en_US
dc.identifier.citation0
dc.identifier.doi10.1109/TGCN.2023.3347276
dc.identifier.endpage644en_US
dc.identifier.issn2473-2400
dc.identifier.issue2en_US
dc.identifier.scopus2-s2.0-85181573774
dc.identifier.scopusqualityQ1
dc.identifier.startpage635en_US
dc.identifier.urihttps://doi.org/10.1109/TGCN.2023.3347276
dc.identifier.urihttps://hdl.handle.net/20.500.12469/5749
dc.identifier.volume8en_US
dc.identifier.wosWOS:001230177900019
dc.identifier.wosqualityN/A
dc.institutionauthorJafari Navimipour, Nima
dc.language.isoenen_US
dc.publisherIeee-inst Electrical Electronics Engineers incen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/closedAccessen_US
dc.subjectSparksen_US
dc.subjectYarnen_US
dc.subjectTask analysisen_US
dc.subjectResource managementen_US
dc.subjectEnergy efficiencyen_US
dc.subjectEnergy consumptionen_US
dc.subjectClustering algorithmsen_US
dc.subjectDistributed computingen_US
dc.subjectenergy managementen_US
dc.subjectresource managementen_US
dc.subjectschedulingen_US
dc.titleAn Energy-Aware Resource Management Strategy Based on Spark and YARN in Heterogeneous Environmentsen_US
dc.typeArticleen_US
dspace.entity.typePublication
relation.isAuthorOfPublication0fb3c7a0-c005-4e5f-a9ae-bb163df2df8e
relation.isAuthorOfPublication.latestForDiscovery0fb3c7a0-c005-4e5f-a9ae-bb163df2df8e

Files