Cloud Computing for Education:
A Systematic Mapping Study
Maria Teresa Baldassarre, Danilo Caivano, Giovanni Dimauro , Enrica Gentile, and Giuseppe Visaggio
Abstract—This paper examines the cloud computing for
education (CCE) literature, and analyzes if the research is devel-oping scientifically with adequate empirical validation. All aspects of empirical investigations covered in the literature are shown as weak, hence, the necessary scientific development of CCE requires extending its scope of interest, and involving schol-ars synergistically to create and maintain a “common research agenda.”
Background: A need to develop research on CCE has been
recognized, and considerable efforts made to create an accurate understanding of the development of its scope of interest, in terms of supporting pedagogical developments and processes for better quality of studies.
Research Questions: This paper has three main aims: 1) to
evaluate the scope of interest in the literature for CCE with specific reference to pedagogy and educational processes; 2) to analyze the characteristics of papers, specifically empirical stud-ies, from the various points of view of the daily improvement activities of teachers and learners at all levels of education; and 3) to identify eventual research gaps to consider and stimulate new topics or further investigations.
Methodology: This systematic mapping study review followed
a rigorous, replicable process to collect and analyze representative studies of CCE.
Findings: Differences are found across geographic areas in
applying CCE infrastructure and technologies in educational institutions; few studies address CCE’s impact on pedagogic processes. The scope of interest in CCE is only partially covered; with empirical research being very shallow. Suggestions are made for more effective research on concerning the production and use of content.
Index Terms—Cloud computing, computer-based instruction,
educational technology, e-learning, instructional methods, learn-ing environment, systematic mapplearn-ing study, technology-enhanced learning.
I. INTRODUCTION
E
-LEARNING (electronic learning) is an active field in the applied computer science domain, enriched by recent information technology innovations. A more common term is Technology Enhanced Learning (TEL) [1]; newly-emerged technologies have led to new forms of TEL, such as m-learning (mobile learning), a service that typically combines mobile and wireless technologies to provide learning instruments and Manuscript received June 1, 2016; revised May 25, 2017 and October 24, 2017; accepted January 4, 2018. Date of publication February 6, 2018; date of current version August 2, 2018. (Corresponding author: Giovanni Dimauro.) The authors are with the Dipartimento di Informatica, Università degli Studi di Bari, 70125 Bari, Italy (e-mail: giovanni.dimauro@uniba.it).Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TE.2018.2796558
educational resources, making lessons available and accessi-ble to anyone everywhere through mobile devices [2]. Another example of TEL is v-learning (virtual learning), providing rich learning resources in a unique, easily accessible virtual environment [3]. E-learning services make use of intensive computing scenarios (e.g., virtual worlds, simulations, video streaming, data analysis, empirical experimentation) and large-scale cooperation for content production such as Massive Open Online Courses (MOOC).
Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (such as networks, servers, storage, appli-cations and services) that can be rapidly provisioned and released with a minimum of management effort and interaction with the supplier [4], [5]. Based on [5], cloud computing should present five essential characteristics: (1) on-demand self-service, allowing consumers to manage their own virtual resources without interacting with the supplier, with broad network access (e.g., the Internet) via heterogeneous client platforms such as smartphones, tablets and PCs; (2) resource pooling, making resources accessible to multiple producers and to any authorized consumer; (3) rapid allocation, provid-ing resources quickly and releasprovid-ing them immediately after use; (4) measured service, automatically controlling the pro-vision of services and (5) metric templates to optimize and measure (in terms of visibility and payment) service provision for both the supplier and the user.
These five features provide many opportunities in educa-tional development [6]. Cloud computing can provide several interesting instruments for both teachers and students, for example, displaying computing resources for lessons and labs on request and according to different user needs. Cloud com-puting facilitates the provision of more flexible courses based on students’ specific needs, when and where they want; it also helps teachers create lesson content that meets their specific requirements [3].
The cost of such resources depends on the quantity and time of use. For example, where teachers need to create a vir-tual computing environment (virvir-tual machine) for a laboratory lesson, some institutions are using cloud computing’s col-laboration and storage services to create a virtual learning environment [3] that would be expensive and time-consuming in real life. Cloud computing facilitates the widespread use of TEL, without need for infrastructure or specific skilled personnel, and within a very short amount of time.
The area of CCE has thus generated great interest among researchers as indicated by numerous studies in [6]–[9], as 0018-9359 c 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
well as the study reported here, whose purpose is to examine the scope of interest in CCE in the literature, and whether the research is developing scientifically with adequate empirical validation.
This paper investigates the contributions of studies on the impact of cloud computing on education, analyzing: new ways of teaching and learning; the development of new pedagogi-cal approaches and other methods of evaluation; educational features derivable from potentially offering more material to a wider range of learners in a greater variety of contexts; the transition from teacher-centered courses to a more learner-centered education; and the management of “networked stu-dents” who access an ever-increasing supply of resources to put theoretical concepts into practice [6], [10], [11]. Gaps between the potential areas of interest and those dealt with in the literature, and/or in the empirical validation of the research results, would delay the effective application of cloud com-puting’s potential in real educational processes. Identifying these gaps will point to new and more effecting research paths necessary to develop the state of the art.
For a wide overview of the CCE research area, this work used the systematic mapping study (SMS) approach [12] to collect data and interpret results on research/development alignment, CCE’s scope of interest, and empirical evidence collected in the CCE literature. The results of this SMS can identify areas suitable for conducting systematic litera-ture reviews (SLR), and areas where a primary study is more appropriate. An SLR is a means of interpreting available and empirically relevant research on a particular research question or phenomenon of interest [12], [13].
The paper is structured as follows: Section II discusses related work and notes the novel contributions of the present study; Section III describes the mapping study process con-ducted to select and classify papers; Section IV analyzes the results of each research question; Section V discusses the rele-vant findings from the research results; and finally, Section VI draws conclusions and outlines possible research directions.
II. RELATEDWORK
A systematic survey of the literature is a secondary study method that has emerged over the last two decades, and that can be seen as an evolution of research based on pri-mary studies of empirical evidence [14], [15]. One of the first systematic surveys was in medicine, a discipline that is particularly invested in evidence-based research. This more recently inspired the transfer of the need for, and con-cept of, a systematic survey to the domain of informa-tion systems [16] and, in parallel, software engineering [17]. Pickard et al. [18] discuss the need to aggregate the results of empirical primary studies to form a secondary study. Miller [15] addresses the issue of combining pri-mary study results through meta-analysis, and Hayes intro-duces the concept of research synthesis. They insist on the importance of systematic primary studies to build knowledge by synthesizing previous results. A first attempt at a synthesis of primary studies was carried out by Basili et al. [19].
CCE is an interdisciplinary research area involving the fields of computer science and pedagogy [20], both disciplines that can benefit from research results based on empirical evidence. The results of this present study could therefore benefit both communities. Given the relatively young age of this research area, the authors consider it appropriate to start with an SMS, since if the results prove limited or inaccurate, it will not serve to perform any synthesis, as suggested in the references cited. For example, an SLR paper [9] in CCE presents relatively few publications relevant for inclusion in the study (only 27); its findings are a subset of those derivable from an SMS, in general, and it state that it lacks empirical evidence, and thus cannot conduct the meta-analysis distinctive of the SLR.
To the authors’ knowledge, there are just two previous literature reviews on the topic. The first study by Fasihuddin et al. [6] analyzes the state of the art of the research without considering the empirical evidence factor. Because this study did not follow a systematic approach, the repeatability [21] and reliability [22] are low, and it does meet the goal of the research presented here. The second study by Gonzàlez-Martìnez et al. [7] is an SMS that provides interesting results on the advantages and limitations of the use of CCE. Although claiming to use the guidelines in [12], it does not really conform to them, for example, evaluating the quality of papers as is typical of an SLR. Moreover, with the quality measurement parameter as the “relevance” of the contribution for the educational domain, “credibility,” “sound-ness,” and “clarity” are not specified; these measures are then subjectively evaluated by the researchers, so the study is not replicable. It outlines the state of the art but does not analyze to what extent the results are based on empirical evidence. The research presented here overlaps the classification of some research topics, and differs from the papers cited previously in many other aspects.
Finally, in this study, the SMS process is described with rigor, and is therefore replicable. Indeed, if researchers notice any threats in this study, they can vary the SMS process to mit-igate that threat, or can reinforce the results or change/extend the period or purpose of the investigation.
III. RESEARCHMETHOD
This research method applies the guidelines of Kitchenham and Chartres [12] to conduct the SMS.
A. Research Questions
Based on Kitchenham’s article [12], the objective of the present SMS uses the paradigm: Population, Intervention, Output (PIO). Kitchenham and Chartres [12] suggested using a “Comparison” factor, excluded here because, in an SMS, the papers’ contents are only partially analyzed. The papers’ authors barely explain if and how they carried out the compar-ison, so selection of these papers based on such criteria cannot be made.
For specific objectives, the PIO paradigm is defined as:
Population: researchers, teachers and recipients in education
of all levels; continuing education workers interested in using cloud computing; and pedagogical institutions;
Intervention: cloud computing during the entire educational
lifecycle;
Output: any benefits that cloud computing can lead to
in terms of organizations, teaching methods and teaching processes.
Taking the general questions of SMS defined by Arksey and O’Malley [23] and structuring them according to the PIO paradigm used in the present study, the dimensions to be investigated are defined by research questions (see Table I).
B. Research Protocol
1) Research String: To build a research string that
auto-matically ensures the extraction of a pertinent literature study sample for the purpose of this paper, one must know the words used in the literature that express the concepts in the PIO. These words, incrementally obtained through the validation process described in the next paragraph, are:
1) Population: teacher; educator; learner; student; teach-ing; education; learnteach-ing; education courses; educational courses; distance learning; computer-aided instruc-tion; education instituinstruc-tion; e-learning; electronic learn-ing; learning management system; distance education; m-learning; v-learning; technology-enhanced learning; TEL; massive open online courses; MOOC; virtual machine; virtual campus; virtual learning environ-ments; VLE.
2) Intervention: cloud computing; cloud.
3) Output: improving teaching; pedagogical changes; global teaching; learning anywhere; learning anytime; teaching anywhere; teaching anytime.
The search string is made up of words or dimensions linked to “OR” and words or dimensions linked to “AND”.
2) Research Strategy: Before performing an automatic
extraction across the entire period of time considered, a search string must be built that ensures the extraction of representa-tive studies in the entire literature concerning CCE. To do this, the technique proposed by Zhang et al. [24], Fig. 1, was used.
Sensitivity is obtained through the following procedure:
Sensitivity or Recall = NSRR/NSRT;
NSRR: Number of Relevant Studies Retrieved, number of relevant studies identified with respect to the topic being investigated, automatically extracted;
NSRT: Total Number of Relevant Studies, total number of studies retrieved manually.
The baseline to meet Sensitivity is 80%.
3) Generation and Validation of the String: The search
string validation examined the period between Nov. 2014 -Jan. 2015 (a three-month mapping study).
First, a manual search using the “snowball technique” was carried out; this method of searching for relevant liter-ature ensures greater completeness of the automatic search. References were identified and collected from each paper of interest, then the analysis of these references led to finding of other relevant papers on the same topic. Then the articles that were cited in each paper of interest were searched, and the
TABLE I RESEARCHQUESTIONS
most significant were individualized using specialized sources for manual searches, such as:
1) IEEE TRANSACTIONS ONEDUCATION;
2) Education and Information Technologies, Springer;
3) IEEE TRANSACTIONS ON LEARNING
TECHNOLOGIES;
4) Global Engineering Education Conference (EDUCON). A total of 43 papers was selected during the period of inves-tigation. Then, an automatic search was also carried out by using search engines related to sources, such as:
1) IEEE (http://ieeexplore.ieee.org/Xplore/home.jsp) 2) ACM (http://dl.acm.org/)
3) Springer (http://link.springer.com/)
4) Science Direct (http://www.sciencedirect.com/). The detailed results are reported in http:// serlab.di.uniba.it/files/Cloud_Computing_for_Education.pdf.
An automatic search was also run using search engines. With the help of search strings containing the most relevant keywords (as defined by the authors), an initial quantity of papers was extracted: extracted papers = 27, sensibility ∼
Fig. 1. Validation of the research string.
63%. The analysis of this first mining enriched the list of words and led to a second extraction resulting in: extracted papers = 32, sensibility ∼ 74%. The study of this sec-ond mining further refined the research string, and data were extracted for a third time: extracted papers= 38, sensibility
∼ 88%. With this last extraction, the string was considered
final. The terms that make up the string have already been listed in Section III-B1.
4) Randomly Assigning Papers to Author Reviewers: The
authors of this paper assumed the roles of reviewers, evaluat-ing each selected publication with respect to the inclusion or exclusion criteria and with respect to the keyword extraction. Each paper received three reviews, and each of the authors were randomly assigned the same number of papers to revise. In total, each author reviewed 348 publications.
5) Screening Relevant Papers for Inclusion and Exclusion:
The screening process reported the most appropriate papers for the mapping study through the inclusion and exclusion criteria provided by the protocol. Thus, journal papers, con-ference and co-located event publications and technical reports were included. When multiple publications dealt with the same topic, the most recent ones were taken. The language consid-ered was English. The data collection period was January 2012 - December 2016.
Studies that did not clearly report results, and papers avail-able only through abstracts or presentations were excluded; studies relevant to cloud computing but with no references to education were also discarded.
For each paper, the three reviewers expressed their opinions for inclusion or exclusion. If they disagreed, they discussed until reaching agreement. If no agreement was achieved within the time limit (30 minutes), the paper was accepted or rejected by majority. TableIIbelow reports the numbers of publications selected and included in this study.
6) Keywords: The reviewers then extracted the keywords
from each paper, paying particular attention to words useful
TABLE II
PAPERSSELECTED ANDINCLUDED INTHISSTUDY
for building classification schemes and helpful for maps of the research questions. They mined the keywords from the abstracts and, in some cases, from the introductions of the papers, if the abstract was inadequate. If necessary, con-clusions were also analyzed. If, after reading the abstracts, introductions and conclusions, a reviewer had not been able to extract the keywords to classify the paper and answer the research questions, the paper would have been automatically excluded, as per the protocol, but this did not happen. A total of 580 papers was classified.
The keywords extracted by each reviewer were compared and evaluated to achieve unanimous agreement. Again, if agreement was not reached within 30 minutes, a majority vote was used. Each newly extracted keyword was then associated with a meaning, and classified with other keywords having the same meaning and extracted from previous papers.
If a new keyword (Ki) presented overlapping semantics with a previous keyword (Kj), a better specification of Ki and Kj was created to eliminate the overlap. If the semantics of Kj needed to be modified, all the papers previously classified with the same keyword underwent review to align the clas-sification to the new updated semantics. This reclasclas-sification was performed by the reviewers who initially classified the keywords.
If a paper, was characterized by more than one keyword for a dimension, it was assigned to multiple classes. A keyword list was produced for each research question that includes all keywords identified by reviewers for all papers, as well as their classification, see Section IV-D.
IV. RESULTS
This section presents and discusses the findings and results obtained by answering the six questions or “dimensions” listed in Section III-A.
A. Temporal and Geographical Distribution of the Publications
For both distributions, the classification of the papers was objective, so interaction between reviewers was necessary to verify the extracted data. Each paper was included in one class. Fig. 2 shows how the number of CCE papers published annually varied from 2012 to 2016.
A significant number of CCE studies has been published since 2012 (133 papers), although this seems to have decreased over 2015/16 (113/79 papers). This decrease could be due to
Fig. 2. Numbers of CCE papers published annually 2012-16, showing decrease in frequency in 2015/6.
Fig. 3. Top ten productive countries.
there being few recent significant scientific achievements in CCE, since cloud computing is maturing in its commercial use. For the geographical distribution of the CCE papers, the graph in Fig. 3 represents the top ten countries where most of these publications were written, and the number of papers published: each paper is classified as a product of the nation of the authors’ institution. In the case of multiple institution nationalities, the paper is classified as belonging to the country with the most authors or in that of the first author in the case of no majority. Overall, 25 publications have authors of different nationality, so only these papers were classified by majority.
As can be noticed, China and India are produced the most where CCE papers (112 and 60 papers, respectively). A rea-son for this could be that, being large territories and the world’s two fastest growing major economies, these nations have greater needs with regards to education with TEL and, in particular, cloud computing. The next most productive country is the United States (45 publications). This confirms American aspiration to gain a competitive advantage in the most inno-vative areas of technology. Spain is the first European country classified (26 papers published), but at a low position. Only two other European countries are in the list, the U.K. and Germany.
B. Stakeholders
Fig.4shows the distribution of the publications with respect to the research recipients.
As expected, educational institutions represent the large majority of stakeholders using CCE (375 papers). Teachers and students using CCE autonomously represent other recipients,
Fig. 4. Distribution of stakeholders indicated in the studies.
Fig. 5. Distribution of papers by educational level of interest.
but at a lower intensity (170 and 189 papers, respectively). Very few publications in CCE (28 papers) were used by com-panies: some as producers of contents and instruments for digital education, others as users of these tools to empower their employees to promote innovation and development.
Fig.5gives the distribution of papers by level of education; even here, some papers belong to many classes, being related to more than one level of education.
Fig. 5 shows that, of the various traditional institutions, universities are the main users of CCE, with 451 papers (com-pare that to primary schools with only 31), perhaps because the majority of active researchers analyze problems more eas-ily in academic fields, identifying the training needs of the population, as well as instruments and experimental topics.
Continuous education is rarely mentioned in scientific research papers. This, and the low number of industrial papers, suggests that the various needs for worker education are scarcely considered. Consequently, few contents, methods and services can be used for education in industry and, in general, for continuous education.
C. Topics of Interest
Fig.6illustrates the results of key wording for the research question “topics of interest.” Studies often address vari-ous topics, so many papers are classified in many classes. The topics extracted were: Sustainable e-Learning, Security,
Augmented Reality, Curriculum Profiling, Interoperability, Technology Enhanced Learning, Innovative Education and HwSw Infrastructures, Fig. 6. The graph shows that the topic of most interest for researchers is Infrastructures for
HwSw (268 papers), which includes the deployment model and
services provided by cloud computing. Security and
interop-erability (60 papers in total) contains general issues for cloud
computing, whatever its scope, and have no specific educa-tional characteristics. The other categories are more relevant to CCE. The topic of least interest is Sustainable Learning (14 papers): this category contains papers on how cloud com-puting overcomes the physical barriers of schools, and how
Fig. 6. Topics investigated in published papers.
Fig. 7. Distribution of the most-studied effects.
education can be delivered to remote sites, with benefits for energy and environmental balance.
Innovative Education covers a class of papers that relate
to new models and learning environments, as well as their effectiveness in transferring knowledge, and students’ abilities to learn via these new methods.
TEL includes papers on to the most advanced new
struc-tures of digital education (e.g., m-learning, Web, distributed communities) acquired on cloud computing-based platforms.
Augmented reality includes the use of this technology in cloud computing education. Finally, curriculum profiling indicates new educational profiles based on cloud computing. Note that many papers study education on cloud comput-ing (328), rather than cloud computcomput-ing-based education (449).
D. Areas of Interest of the Studies
Fourteen keywords were extracted from the papers to express their main areas of interest.
Fig.7 shows that the majority of the studies focus on Cost
Saving (with 169 papers); cost reduction and other economic
issues related to CCE are among the advantages obtained from the study of these papers.
New Learning Environments is the second most studied
effect with 148 papers: these environments include cloud-based e-learning models as well as proposals, solutions, architectures, and platforms based on cloud technologies to
integrate, improve and share resources, and to teach educa-tional activities.
Collaboration platforms is next with 127 papers; virtual
collaborative environments can improve efficacy of distance educational processes by leveraging cloud technologies.
Cloud Based m-Learning has 101 papers: this includes
aspects related to mobile learning and the potential of cloud computing to solve the lack of specific educational resources, to improve learning, and to operate scaleably at low risk and low cost.
Virtual Laboratories, with 94 papers, treats systems that
cre-ate and offer remote virtual laboratories to improve students’ experiments and act as locations for practical exercises.
Content Sharing (82 papers) covers specific issues related to
the “federation of contents” that various universities or schools share to create a wide range of shared courses and information.
Innovation of Contents (75 papers) pertains to cloud-based
solutions to improving content innovation features (such as student-oriented content) more efficiently and with diverse characteristics (such as the detection of attention disorders or of eventual content problems) and the integration of papers from various sources (TV, radio, the Internet).
Campus Management (59 papers) concerns the
integra-tion and/or migraintegra-tion of IT or non-IT components in IaaS (Infrastructure as a Service) systems, and eventual solutions to making universities’ global resources more accessible to a wider public, as well as the federation of different and dis-tant digital campuses with the main aim of sharing resources and reducing costs.
Security (46 papers) includes papers aimed at
improv-ing end-to-end security to preserve user privacy and ensure data integrity in stored and/or shared information.
Digital Divide (46 papers) refers to reducing gaps in
educa-tion and training in specific geographical areas without access to digital information and communication technology.
High Performance (43 papers) covers applications designed
to perform complex initiatives that require large computing capacity over short periods of time.
Storage Sharing (42 papers) includes specific proposals for
saving teaching modules, support material and so on that require large amounts of storage.
Student Response Systems (SRS) – Evaluation (29 papers):
refers to students’ learning experience during student/teacher interactions and/or to improving the quality and timeliness of continuous evaluations during educational activities.
Library Management is the least studied effect (with only
seven papers). Libraries are usually considered as one of the main foci in schools and universities, as being a “collector of science” and a place of study and exchange of experi-ences. This aspect includes the usefulness and effectiveness of systems implemented on cloud architectures for managing both traditional and digital libraries.
To summarize, many papers explore the impact and results of the use of cloud computing in a general way, and do not consider its specific advantages for education. Cost saving, security and storage sharing have proven advantageous across the use of cloud computing, not just in education. Some papers deal with significant topics specific to cloud computing for
TABLE III
TAXONOMY OFTYPES OFPUBLICATION
education (see virtual laboratories or content sharing), or with cloud computing innovations of both content and approaches to education. The literature also contains various results on cloud computing being used to improve knowledge growth, some involving educational administration.
The cloud computing dimension has the highest number of multiple classifications of publications; these papers are likely to discuss multiple results and may lack deep analysis of all aspects treated. Consequently, the impact and applicability of the results may be poor.
E. Type of Studies
Having identified the number of CCE papers with empirical evidence in the literature, the papers were classified according to a taxonomy introduced in [16] and adapted to the aims of this research, TableIII. The classes of this dimension were not extracted by keywording, which is why all the papers have a unique classification.
Fig. 8. Distribution of papers by classification type.
Fig. 9. Distribution of where studies are published.
Fig.8reports the distribution of the papers in the classes in Table III. Many papers (196) propose solutions; fewer (113) contain any type of empirical evaluation, and even fewer (51) empirically validate their proposals. Of the other classes, sev-eral papers are limited to providing opinions, experiences or conceptual scenarios, and are thus very far from propos-ing innovations that can be implemented in real educational processes with low risk of failure.
Those papers collecting empirical evidence do so in vari-ous fields, so the results are usually easier to transfer to real educational processes, but without controlled experiments it is impossible to generalize their results and formulate theories that can be used in different contexts. Indeed, generalization is a necessary precursor to producing theories, and for the scientific growth of the knowledge domain being investigated. The papers’ distribution and editorial classification, Fig.9, shows that the literature concentrates on conferences.
The predominance of CCE conference papers implies that the published results lack empirical evidence, that studies are still at an initial phase, or that the results are not interesting or significant enough to be published in a journal. This confirms not only the lack of empirical evidence for the efficacy of cloud computing solutions in education, but also confirms that still-immature solutions are presented in the literature for adoption in educational processes.
F. Solution Status
This dimension only includes papers that present solu-tions, and excludes, for example, philosophical or disclosure
Fig. 10. Distribution of solution types presented.
Fig. 11. Study Type vs. Output.
papers. Their distribution, Fig. 10, shows that most (263) are “designs”, that is, proposed but not-implemented technolo-gies, without even a prototype version. Fewer papers (146) are “experimental”, proposing solutions with empirical valida-tion, usually presenting prototype tools. Sixty-nine papers are “prototype”, presenting tools that were implemented but only validated on toy examples. Only a few papers report “work-ing” cases, that are beyond laboratory experiment status and currently adopted, even if just at an exploratory level.
G. Mapping
Each main dimension class was then mapped against cate-gories. Three maps are shown here: study type vs. output (i.e., benefits of cloud computing, see Section III-A) Fig.11; study type vs. topic of publication, Fig.12; and study type vs. level of education, Fig. 13.
In the Empirical Evaluation category in the first map, Fig. 11, the numbers of papers with most relevance to education and with the highest empirical evidence are (in decreasing order) virtual laboratories, collaboration platforms, m-learning platforms and new learning contexts. Empirical investigations primarily focus on educational aspects more influenced by cloud computing. In the Empirical Evaluation category, 243 elements were obtained from the 113 papers included, meaning that every paper carried out an Empirical Evaluation on more than one topic; experiments were carried out with multiple goals, so probably with a low confidence for each result, reducing the likelihood of their being applied in real educational processes. Similarly, the Empirical Validation
Fig. 12. Study type vs. publication topic.
Fig. 13. Study Type vs. Level of Education.
category has 92 elements generated from 51 papers, leading to the same conclusion that results reported in the current literature do not have a high confidence level.
In the map of study type vs. publication topic, Fig.12, many studies collecting empirical evidence focus on hardware and software infrastructures, general topics in cloud computing. However, some empirical evidence in specific topics related to the adoption of CCE are illustrated in studies of innovative teaching and TEL. The map shows that study topics are dis-persed: 161 elements distributed in 113 papers with empirical evaluation and 73 topics in 51 papers with empirical validation. Presumably the individual topics are not treated exhaustively in each paper.
In the map of study type vs level of education, Fig. 13, papers showing many types of empirical evidence are mainly university studies, with some pertaining to secondary schools. This is to be expected as most researchers are essentially
academics whose experimental fields are in university courses or in secondary school courses with which universities collabo-rate, which why there are more empirical evaluations than val-idations. Indeed, university professors can experiment directly with their solutions for educational processes during their classes. Other types of publications also have a higher concen-tration on universities and secondary schools, confirming the previous proposition. Finally, the map again shows a disper-sion, although to a lesser extent: 107 elements in 76 papers for Empirical Evaluation and 23 elements in 19 papers for Empirical Validation. As discussed above, dispersion indicates the studies considered each topic only superficially, so results probably have low credibility for their stakeholders.
V. DISCUSSION
This paper has analyzed how CCE research develops, con-sidered the results from various points of view, identified some gaps in the literature, and suggested ideas for future research that could scientifically strengthen this area of study by apply-ing these results to real educational processes. By answerapply-ing the research questions of TableIand examining the data maps generated, the study came to the following findings:
RQ1). The development of CCE has kept pace with the development of cloud computing. This trend is not uniform all geographical areas, seeming to increase in the most disadvan-taged areas, with the least developed countries more rapidly adopting CCE to quickly improve services to students and the administration at relatively low cost, as clearly demonstrated by Joshi [8] in India.
RQ2). Most studies focus on the university and high school levels, with very few addressing other levels of education, particularly the cases of continuing education for workers.
RQ3). For the cloud computing topics investigated in the various papers, technological aspects seemed to prevail; the impact of cloud computing on pedagogy and its innovations for teaching and learning processes have been neglected.
RQ4). There is a predominance of papers dealing with the financial advantages of cloud computing in general, as opposed to CCE specifically. They confirm the development of specific cloud computing technological achievements for e-learning and management of educational institutions, but give no results on how cloud computing could support education and improve educational processes, a deficiency that should be resolved.
RQ5). Most of the publications are of the type “not empiri-cal” rather than “empirical.” Therefore, even if the topics and results are treated and analyzed deeply, they lack empirical evidence, so the validity of the results is questionable. Very few studies quantitatively analyze CCE’s costs, benefits and drawbacks, and their solutions are essentially in the design phase. Some papers deal with the prototype stage, confirming the hypothesis of the scientific immaturity of publications in this area.
RQ6). By far the most common publication channel for CCE is the conference meeting, a forum generally used to present relevant results for consideration, evaluation and feedback from experts in the related scientific field to improve the research presented. These studies are therefore expected to be, to some degree, scientifically immature. The mapping
data reveals that many studies perform empirical research on many topics, rather than a single topic, suggesting that the empirical evidence presented can often be inaccurate and/or unreliable.
At this point, the authors believe it possible to suggest some guidelines for future studies in CCE. The analysis confirms the need to intensify research in cloud computing in a more bal-anced way, that is, at all levels of education rather than just one or two levels. If cloud computing is not used at all levels of education, the benefits of this new technology will not be fully realized. Indeed, without synergy of innovation at various levels of education, students would encounter unknown teach-ing approaches when transitionteach-ing between grade levels. This would generate confusion and adversely impact the learning curve they could achieve with innovative technologies.
CCE research should give more attention to continuing education, which will be increasingly demanded as compa-nies need continuous innovation due to competitive pressure, necessitating skill upgrades for their human capital. Cloud Computing and related technologies can have a high impact in continuous education through its economic and technical advantages, and greater flexibility for education.
Empirical investigations are already being developed in other scientific areas such as computing engineering. In CCE, the authors believe the scope of interest should include the following mid-term research initiatives:
1) Improving the quality of empirical evidence, especially on topics on innovative teaching, the profiling of new curricula and TEL. Innovative content, currently being developed (e.g., MOOCs) and available anywhere at any time thanks to cloud solutions, should be considered, because they can radically change educational processes. In this type of research, the cloud should become a cen-tral point for education. Studies should compare the use of established or developing educational technol-ogy with and without cloud computing, and validate its advantages and disadvantages.
2) Investigations should cover all empirical study methods, from controlled experiments to case studies and action research, providing empirical evidence of the technolo-gies within real educational processes. Depending on the researcher, she may generalize a theory, create or con-firm a local theory, or even be the principal actor for changing a community keeping in mind the innovation factor [16]. This would considerably increase scientific knowledge of CCE.
3) The recipients’ “experience” should be considered, espe-cially when exposed to educational innovations focused on cloud computing in different contexts and in dif-ferent educational paths. This could indicate when and how to adopt cloud computing to make education most effective.
The evident need for scientific development of CCE means extending the scope of interest of this research area; there-fore, it seems necessary to involve scholars synergistically to constitute and continuously update a “common research agenda” [20]. This would speed the achievement of goals and meaningful research results.
The most significant threat to the findings of this study is that the sample of primary studies selected for this analysis may be incomplete. Extraction of papers in the literature to moderate this threat cannot be ensured, as required by the SMS method. It is the authors themselves who state that the papers chosen were representative of the entire literature, and that any papers not identified by this study would not significantly change the results, even if other researchers were to replicate this study at low cost and quickly.
Mapping studies are time-limited, so it is generally neces-sary to repeat the mapping periodically to check if factors have changed and how they are evolving.
An advantage of SMS is that the maps also indicate the best perspective from which to do an SLR (requiring rela-tively higher cost and time), and minimizes the risk of waste because of the lack of significant empirical investigations. This study, in particular, indicates that the most appropriate CCE topics on which to conduct an SLR are “Innovative Education” and “TEL”.
VI. CONCLUSION
This mapping study identified significant interest in CCE from the high number of papers in the literature over the three-year period analyzed. Research in CCE is developing in line with research in cloud computing, but an important step would be for CCE to grow more in developing countries’ universi-ties and higher educational institutions, especially on topics of general interest related to cloud computing (e.g., cost saving, security, high performance).
The scope of interest in CCE in the literature mainly focuses on the technological aspects, instead of on the impact of cloud computing on pedagogy and the innovations that it could bring to teaching and learning processes. Many papers explore the impact and the results of using cloud computing in a general way, rather than considering its specificity for education; they focus on proven advantages of cloud computing—such as cost saving, security and storage sharing—rather than on how it applies to education.
The literature reports partial results as suggested by the cat-egories of output and where studies are published. Specific empirical research on the impact of cloud computing on edu-cational processes is lacking, even in validating the most commonly claimed advantages of cloud computing (e.g., cost saving, high performance, teaching/learning anywhere and anytime) for educational institutions, teachers and learners.
The main gaps to bridge are to balance research at all levels of education, to increase interest in continuing education for workers, and to enlarge the field of interest to specific factors of cloud computing in pedagogy and educational processes. Care should be taken to introduce the research into new tech-nologies in educational organizations and processes. Empirical investigations in CCE must be intensified, to facilitate the adoption of CCE in different fields and to demonstrate the multidisciplinary theories of cloud computing (see computing engineering, pedagogy and education). Development of theo-ries requires scientific and technological development of CCE. The authors hope for a closer collaboration between CCE researchers, and the constitution of a “common research
agenda,” as is occurring in other areas of computing engi-neering.
REFERENCES
[1] L. Johnson, S. Adams, and M. Cummins, “NMC horizon report: 2012 Higher education,” New Media Consortium, Austin, TX, USA, Rep., 2012. [Online]. Available: https://www.nmc.org/publication/nmc-horizon-report-2012-higher-ed-edition/
[2] T. Georgiev, E. Georgieva, and A. Smrikarov, “M-learning: A new stage of e-learning,” in Proc. 5th Int. Conf. Comput. Syst. Technol., Ruse, Bulgaria, 2004, pp. 1–4.
[3] N. Sultan, “Cloud computing for education: A new dawn?” Int. J. Inf.
Manag., vol. 30, no. 2, pp. 109–116, 2010.
[4] B. Furht and A. Escalante, Eds.,Handbook of Cloud Computing. New York, NY, USA: Springer, 2010, doi:10.1007/978-1-4419-6524-0. [5] P. Mell and T. Grance, The NIST Definition of Cloud Computing, NIST,
Gaithersburg, MD, USA, 2011.
[6] H. Fasihuddin, G. Skinner, and R. Athauda, “A holistic review of cloud-based e-learning system,” in Proc. IEEE Int. Conf. Teaching Assess.
Learn. Eng. (TALE), 2012, pp. H1C-6–H1C-11.
[7] J. A. Gonzàlez-Martìnez, M. L. Bote-Lorenzo, E. Gòmez-Sànchez, and R. Cano-Parra, “Cloud Computing and education: A state-of-the-art survey,” Comput. Educ., vol. 80, pp. 132–151, Jan. 2015.
[8] M. Joshi, “Emerging trends of cloud computing in education method-ologies: A review,” Int. J. Adv. Res. Comput. Sci. Softw. Eng., vol. 5, no. 12, pp. 1–4, Dec. 2015.
[9] M. S. Ibrahim, N. Salleh, and S. Misra, “Empirical studies of cloud com-puting in education: A systematic literature review,” in Computational
Science and Its Applications—ICCSA 2015 (LNCS 9158), O. Gervasi et al., Eds. Cham, Switzerland: Springer, 2015.
[10] R. Boyatt and J. Sinclair, “Navigating the educational cloud,” in Proc.
Workshop Learn. Technol. Educ. Cloud, vol. 173, 2012, pp. 179–191.
[11] M. S. Ibrahim, N. Salleh, and S. Misra, “Empirical studies of cloud computing in education: A systematic literature review,” in Proc. Int.
Conf. Comput. Sci. Appl., Jun. 2015, pp. 725–737.
[12] B. Kitchenham and S. Chartres, “Guidelines for performing system-atic literature reviews in software engineering,” Keele Univ., Newcastle, U.K., and Durham Univ., Durham, U.K., Rep. EBSE-2007-01, 2007.
[13] B. A. Kitchenham, T. Dybå, and M. Jørgensen, “Evidence-based soft-ware engineering,” in Proc. 26th IEEE Int. Conf. Softw. Eng., 2004, pp. 273–281.
[14] W. Hayes, “Research synthesis in software engineering: A case for meta-analysis,” in Proc. 6th IEEE Int. Softw. Metrics Symp., Boca Raton, FL, USA, 1999, pp. 143–151.
[15] J. Miller, “Can results from software engineering experiments be safely combined?” in Proc. IEEE 6th Int. Symp. Softw. Metrics, Boca Raton, FL, USA, 1999, pp. 152–158.
[16] R. Wieringa, N. Maiden, N. Mead, and C. Rolland, “Requirements engineering paper classification and evaluation criteria: A proposal and a discussion,” Requirement Eng., vol. 11, no. 1, pp. 102–107, 2006. [17] D. I. K. Sjoberg, T. Dybå, and M. Jørgensen, “The future of empirical
methods in software engineering research,” in Proc. IEEE Future Softw.
Eng. (FOSE), May 2007, pp. 358–378.
[18] L. M. Pickard, B. A. Kitchenham, and P. W. Jones, “Combining empiri-cal results in software engineering,” Inf. Softw. Technol., vol. 40, no. 14, pp. 811–821, 1998.
[19] V. R. Basili, F. Shull, and F. Lanubile, “Building knowledge through families of experiments,” IEEE Trans. Softw. Eng., vol. 25, no. 4, pp. 456–473, Jul./Aug. 1999.
[20] N. Sclater, “Cloud computing in education,” UNESCO Inst. Inf. Technol. Educ., Moscow, Russia, Rep. IITE/INS/PB/2010/01, 2010. [Online]. Available: http://unesdoc.unesco.org/images/0019/001904/190432e.pdf [21] B. Kitchenham, P. Brereton, Z. Li, D. Budgen, and A. Burn,
“Repeatability of systematic literature reviews,” in Proc. Int. Conf. Eval.
Assess. Softw. Eng. (EASE), Durham, U.K., 2011, pp. 46–55.
[22] S. MacDonell, M. Shepperd, B. Kitchenham, and E. Mendes, “How reliable are systematic reviews in empirical software engineering?” IEEE
Trans. Softw. Eng., vol. 36, no. 5, pp. 676–687, Sep./Oct. 2010.
[23] H. Arksey and L. O’Malley, “Scoping studies: Towards a methodological framework,” Int. J. Soc. Res. Methodol., vol. 8, no. 1, pp. 19–32, 2005. [24] H. Zhang, M. A. Babar, and P. Tell, “Identifying relevant studies in software engineering,” Inf. Softw. Technol., vol. 53, no. 6, pp. 625–637, 2011.
Maria Teresa Baldassarre received a degree (with Hons.) in informatics and the Ph.D. degree from the University of Bari, Italy, where she is currently an Assistant Professor. Her research interests focus on empirical software engineering, harmonization of multiple improvement technologies, quality assessment, and improvement in software. She collaborates in sev-eral research projects and works within small and medium enterprises. She is a Partner of the SER and Practices spin-off company. She currently represents the University of Bari in the International Software Engineering Research Network, and is involved in various program committees related to software engineering and empirical software engineering international conferences.
Danilo Caivano received the graduation and Ph.D. degree from the University of Bari Aldo Moro in 2002, and where he is currently an Assistant Professor, performing research with the Software Engineering Laboratory, Department of Informatics. His research and teaching activities focus on topics related to software engineering with emphasis on project and process manage-ment in collocated and distributed contexts, and on software developmanage-ment, maintenance, and testing. Since 2007, he has been the Chief Executive Officer of SER&Practices, a spin-off company of the University of Bari that he helped start up. He is actively involved in the Project Management Institute, Southern Italy Chapter, and in the International Software Engineering Research Network.
Giovanni Dimauro received the Laurea degree in computer sciences from the Università degli Studi di Bari in 1987, where he is currently an Associate Professor of computer programming and multimedia systems. His research activities described in over 150 papers published in scientific journals, pro-ceedings and books, are currently on e-health, multimedia systems, cloud computing, pattern recognition with applications in medicine such as new diagnosis technology for anemia and in Parkinson disease.
Enrica Gentile received the degree in computer science from the University of Bari in 1986. From 2002, she was a Senior Technician with the Department of Informatics, University of Bari, where, since 2002, she has been an Assistant Professor of informatics. Since 1987, she has also been running the research laboratory LaRTE: LAboratorio di Ricerca sulle Tecnologie dell’Educazione. Her scientific and research activity concerns Web informa-tion systems, educainforma-tion engineering, and e-learning supported by informainforma-tion technologies.
Giuseppe Visaggio was a Full Professor of software engineering until 2017 and the President of the Laurea Degree Courses in informatics held with the Department of Informatics, University of Bari “Aldo Moro,” from 2011 to 2015. His research interests are the production and maintenance of large and ultra large software systems, process-based software development, software based on services and distributed as services, software development based on closed- and open-source components, software engineering education, and empirical software engineering. He has published several papers on the above topics and has been involved in the organization of many international con-ferences. He is actively involved in strengthening the network of relations between the university and enterprises for technology transfer from research laboratories to industry. This type of involvement has led him to establish a university spin-off company “SER&Practices” in the domain of software development. He is the President of the DAISY-Net Competence Centre for ICT transfer toward users and supply enterprises of ICT itself and was the President of the inter-university consortium Universus for higher Post-Laurea and Post-Doctoral education in technology and management.
