Journal Search Engine
Search Advanced Search Adode Reader(link)
Download PDF Export Citaion korean bibliography PMC previewer
ISSN : 2092-674X (Print)
ISSN : 2092-6758 (Online)
Asia-Pacific Collaborative education Journal Vol.6 No.2 pp.69-86
DOI :

Laboratory as a Service: Architecture, Implementation and Experiences1)

Mojca Ciglarič, Andrej Krevl, Milan Jeličič, Andrej Brodnik
Mojca Ciglarič received her PhD, MSc and BSc in computer science from University of Ljubljana, Faculty of Computer and Information Science. She is an Assistant Professor and a head of Computer Communications Laboratory at University of Ljubljana. Her research work is in the area of computer communications and distributed systems.
Andrej Krevl, BSc, works as a teaching
assistant at University of Ljubljana.
Milan Jeličič works at Nil d.o.o. as a
head of remote laboratory activities and
related research.
Andrej Brodnik received his PhD and BSc
in computer science from University of
Waterloo, Ontario, Canada and University
of Ljubljana, respectively. His main
interests are succinct data structures,
ubiquitous systems, and efficient algorithms
in network engineering. He is one of the
authors of Lulea algorithm. Presently, he
teaches at University of Ljubljana.
Received Date: Oct. 17, 2010, Revision received Date: Dec. 15, 2010, Accepted Date: Dec. 20, 2010

Abstract

In Computer Science study, practicalexperience is gained in the computerlaboratory, however the laboratory fundingand maintenance require huge resources.This is the main reason why virtualcomputing laboratory implementations inacademic environments have increased ourmajor interest.At University of Ljubljana, we are theonly higher education institution usingvirtual laboratories as an integral part ofany Computer Science courses not only inSlovenia but also on the Balkans. Initiallywe have built our own virtual laboratory.Later, we merged our implementation withNCSU VCL and integrated the new virtuallaboratory with learning managementsystem and identity management system.We also plan to integrate it with ourstudent information system.In this paper we present our currentimplementation and future plans, wediscuss how this can improve the learningoutcomes and present our experiences andresults gained in the spring semester of2010 when used by more than 300students who were turned out to havelittle significant technical difficulties. Inorder to complete two assignments, thestudents made 8.9 reservations on average,while the minimum of four was needed toaccomplish the obligations. 27.5% of thestudents needed more than 10 reservations(16.8 on average) and are responsible for52% of total reservations. Furtherresearch is needed to find true causes forthat fact. Most popular hour of day was 3pm and 10 pm, while the least popularwas 6 in the morning, when noreservations have been made in the wholeobserved period of three weeks. Researchalso shows that the students tend topostpone their duties until just before thedeadline. The findings enhance ourmotivation on further research related tokeeping students engaged and activethrough the course.

0081-02-0006-0002-5.pdf1.33MB

I. Introduction and Related Work

 One of the most distinguishing properties of science and engineering studies is laboratory work. It turns out that students after two weeks period remember on average 10% only of what they have read and up to 90% of what they have done practically in the lab. The paper describes how a group of researchers and teachers at University of Ljubljana managed to put additional emphasize on laboratory work in order to enable higher knowledge retention. Such a decision has its consequences, particularly higher demand of the resources – infrastructure (computers and other equipment) as well as people (teaching and laboratory assistants). Furthermore, besides increasing the amount of time that students are able to spend in a laboratory, it is highly desirable that this time becomes more flexible. Our goal was actually to introduce a shift in time and space for our students’ attendance in laboratories.

 In the same time when we were planning the described changes, a major change in higher education system in Slovenia was underway. Like other European countries, Slovenia introduced a new higher education scheme known as Bologna system. At the University of Ljubljana, Computer Science program was changed together with other programs. For the context of this paper, the most drastic change was shift of the course on Computer Networks from the second to the first year in university study program and from third to the first year in professional study program. The result was a double cohort of students in a study year 2009/2010, as old and new programs were running in parallel for old and new students. At the end, we were faced with a problem where we wanted simultaneously to increase the laboratory time and the number of students spending their time in laboratories with no extra investment in the buildings and hiring new people. In other words, we wanted to realize more than doubled amount of laboratory training for our students, with no increase in resources. To achieve this ambitious goal, we first needed to browse through existing body of knowledge to find out what has been done before and what experiences did other researchers and teachers gain during similar projects.

 Many researchers describe their specific implementations: Greenhow et al. (Greenhow, 2009) give a general overview of recent teaching issues, while Powell et al. (Powell, 2007) describe a virtual laboratory similar to ours however less integrated with other systems. Gravier et al. (Gravier, 2008) have made a review of different remotely accessed laboratory implementations and they identified a trend in virtual computing laboratories as ongoing mutations – new and new virtual laboratories emerging instead of joining and contributing to existing projects. Further the authors pointed out some future challenges: reusability, interoperability and support consistent with modern learning theories (especially support for collaboration and constructivism), as well as convergence of learning management systems with remote laboratories. Hardaway et al. (Hardaway, 2005) present a different approach to laboratory realization – their team outsourced virtualization infrastructure for university labs. Li (Li, 2009) explores several types of virtualizations used at East Carolina university and their effects – it turns out that no correlation is found between virtualization type and student grades, however subjectively the students preferred more decentralized options (virtual machines running on their own computers).

 Li et al. (Li, 2009a) further describe use of NCSU VCL beside the before mentioned virtualization options and present qualitative student feedback, however the analysis is relatively superficial. Beside this one, several other implementations are based on NCSU VCL implementation (Murphy, 2009; Schaffer, 2009). Anisetti et al. (Anisetti, 2007) describe an implementation of virtual laboratory for networking, based on para-virtualization technique, and shows that it can be used to achieve the same level of students’ performance as those normally obtained when students directly interact with physical networking equipment. Bochicchio et al. (Bochicchio, 2009) suggest a reusable framework for remote access to web-enabled specialized devices, which can be used in remote laboratories. Further, they focus on collaborative work in a virtual laboratory, thus achieving social components and synergy among learners.

 Barros et al. (Barros, 2008) performed an experiment in virtual laboratory, where actual laboratory work was accompanied by pre-lab and post-lab activities, intended for student collaboration activities. Their finding is that the students with more intensive collaboration won significantly higher final grades. Cooper et al. (Cooper, 2009) discuss the rationale for remote laboratories as well as assessment issues, instructional and pedagogical issues and cost benefits. Remote laboratories are similar to virtual laboratories in a way that they enable access to remote equipment; however this equipment is not usually virtually-made as is the case in virtual laboratories. Lowe et al. (Lowe, 2009) present shortcomings of existing remote laboratory implementations and although focused on electro-technical instruments, offer several generally applicable opportunities for improvements mainly in the area of learner communication and collaboration, which also relate to virtual laboratory implementations.

 In the remaining of this paper we describe how we combined our ideas with the findings from other researchers and put them into practice. In section II we overview our virtual laboratory system architecture and identify its basic building blocks. In section III we describe our virtual laboratory implementation from the technical point of view and explain how it is used in a Computer Networking course. In section IV we present the service interfaces offered by virtual laboratory building blocks and a few characteristic scenarios of their use. Since all the virtual laboratory functionality is exposed through standard service interfaces, we believe we justified using the term “Laboratory as a Service” for the SAKE architecture behind our virtual laboratory implementation, following the paradigms behind actual terms like Infrastructure as a Service (IaaS) or Platform as a Service (PaaS). In section V we analyze the nature of typical student’s virtual laboratory access and our experiences gained with the first generation of students who used our virtual laboratory as an integral part of computer networking course. Section VI concludes the work.

II. The Architecture

 The solution we were looking for would belong in the broad area of Information and Communication Technology (ICT) in education. We intentionally moved away from purely ICT supported education without direct interpersonal relations and decided to use a so called “blended learning” concept that keeps benefits of face-to-face teaching and class interaction on one hand and on the other hand brings benefits of ICT supported environment (Garrison, 2008, Singh, 2003, Lapuh Bele 2007). Furthermore, the contemporary blending process does not just blend ICT-based and non-ICT-based environments, but also different aspects of the educational process (e.g. learning and practice or experience-based learning with classical frontal teaching).

 Moreover, to enhance the quality of the learning outcomes we also opted heavily for a so-called constructivist and collaborative approach (Nančovska Šerbec, 2009, Ben-Ari, 2001). The decision for the former was based on the data mentioned in the introduction (success of learning through the work), while the second decision had a double motivation. First, collaborative work is highly recommended in a learning process since social component enhances student motivation and secondly, due to the lack of human resources we also expected to alleviate this problem slightly using a collaborative approach through students helping fellow students.

 So far we were only discussing the objectives we wanted to achieve. In our design of the final system we did not want to repeat the same mistake many other designers of ICT supported learning environment did by simply putting together various building blocks and integrating them into a single system by means of dedicated interfaces. To achieve broader usability we decided to define a general architecture of a learning environment and interfaces among the building blocks first and only after that make choices regarding the appropriate systems to use in place of respective building blocks. In other words, we designed the system in a top-down fashion. Figure1 presents our SAKE architecture (SAKE stands for “Spletna Arhitektura kot učna tehnologija za Konstruktivistično E-učenje” in Slovene - for “Web Architecture as Educational Technology enabling Constructive E-Learning” in English). As we can see, the main system building blocks are :

 o Virtual infrastructure cloud to host virtual machines, with its own management service

 o Learning Management System (LMS)

 o Authentication, Authorization and Identification infrastructure, offering identity management services

 o Existing social networks, used to enhance collaboration

 o Thin clients enabling user access to the virtual laboratory

 We first defined the typical scenarios of use, which are presented in section IV. Then we identified the services, which would implement these scenarios. After that we defined the schemas, describing the data that would be exchanged by means of these services. With these basic specifications in mind, we made a thorough overview of existing candidate systems to be integrated in place of our basic building blocks and made appropriate choices.

III. Virtual Laboratory in Computer Networking Course

 In this section we focus on a virtual laboratory building block (virtual infrastructure cloud) of SAKE architecture as depicted in Figure 1, which will be described from the technical point of view. Our initial set of requirements regarding virtual laboratory to be used in Computer Networking course is quite lengthy. From the pedagogical point of view, virtual laboratory should support not only constructive and experience-based learning, but also collaborative learning, practical hands-on experience, and student creativity. From the teacher’s point of view, automated assessment and integration with learning management system (LMS), enabling collaborative work and social network technologies are required. Technically, virtual laboratory should support basic as well as advanced networking assignments.

Figure 1 SAKE architecture including LMS (learning management system), identification servers, videoconference server, virtual laboratory service, external social networks and (thin) clients

 We have decided to merge our existing virtual laboratory implementation with open source Virtual Computing Lab implementation from North Carolina State University (NCSU VCL) after we have evaluated several implementation possibilities and decided that NCSU VCL is best suited for our needs.

 NCSU VCL consists of a number of virtual machines that users can book in advance. It further provides a remote access service to the reserved virtual computers. The whole service is powered by the Apache Software Foundation's Virtual Computing Lab software (for more details see Schaffer, 2009 and Vouk, 2008). Of the requirements listed above, constructive learning and basic networking assignments could have been possible with VCL as is, but the rest had to be implemented and integrated into our existing framework by ourselves.

 NCSU VCL provides an appropriate technical tool we needed for our SAKE architecture. To completely integrate it into our environment we had to add further support for constructive and experience-based learning, and in particular for collaborative learning. An important functionality in our architecture is assessment of the students’ laboratory work, which can be formative or summative. As shown in Figure 2, depicting assessment-related scenarios, we have included facilities that can support both aspects of assessment: formative assessment is used to give immediate feedback to students and can be used to direct them in further learning activities, as well as summative assessment at the end of learning activities to measure the achievement of learning goals and to certify the final result of learning (Lapuh Bele, 2007).

Figure 2 Support for formative (immediate feedback to student from assessment engine) and summative assessment (final grade) in SAKE architecture.

 Students need opportunities for formative assessment and for getting feedback to develop further skills and concepts. As there are often not enough opportunities to get detailed feedback from teachers, peer-learning settings provide opportunities for additional feedback (for example, by means of discussion within student forum or social networks). Peer assessment (Sluijsmans, 1999) is a process where students consider and specify the level, value or quality of a product or performance of other people in similar situation, usually student within a given class. It also represents an approach to train students how to provide valuable feedback and suggestions for performance improvement on one hand and also how to reflect on their own work on the other hand. Because we wanted to allow students to complete activities in a virtual laboratory according to their own schedule, we cannot provide a continuous presence of teachers or tutors (e.g. during the night). On the other hand, in virtual laboratory the students are supposed to perform many activities by themselves, and hence a feedback of formative assessment is crucial. Formative as well as summative assessment can be automated and implemented as a service of the assessment engine, consisting of either a complex expert system or a few texts parsing scripts or anything in between. As such, assessment engine represents another building block of SAKE architecture. Assessment may also be provided by human tutors and in this case some delay is acceptable. The learning experience is enhanced also by collaborative learning and peer-learning paradigm that was implemented through networks and even public (general) social networks. The results of formative assessment are presented to students in the virtual laboratory, while the results of summative assessment need to be sent to directly into the LMS where they are stored and available to teachers.

 Technically, to use virtual laboratory efficiently in Computer Networking and in Operating Systems courses, it should support basic networking assignments including configuration of network interfaces, inspecting network traffic, setting up routing tables, configuration of various network services like DNS, SMTP, HTTP, etc. Support for IPv6 is also desired.

 NCSU VCL version 2.1 initially only supported Windows based (XP, Server 2003, Vista, Server 2008), and Debian GNU/Linux (including Ubuntu Linux) based virtualized operating systems. We have added support for FreeBSD 8.0 and looked at possibilities to support RT Linux.

 The number of students that took Networking Courses simultaneously was almost 400. To facilitate such a number of students, storing copies of all virtual machines (VM) for all students was not possible. Therefore we needed a new functionality permitting creation of a virtual machine snapshot instead of a complete base image copy. Hence, we implemented and integrated a new provisioning module, based on suggestions found on VCL mailing list. Instead of the default approach, connecting VCL to the NAS storage directly to perform the file system operations, our new module operates through VMware ESXi host relying on VMware's Perl APIs. By this approach, a VM snapshot is created almost instantly and occupies much less storage space since only changes from the base image need to be stored. The new provisioning module can be used directly with the VMware vCenter server. Having all ESXi hosts registered in vCenter, the administration, configuration, debugging and logging are considerably simplified. We have also added support for IPv6 protocol within the assignments and implemented a stress testing tool.

 In the laboratory part of the Computer Networking courses, the students have access to three virtual machines (Linux, FreeBSD and Windows) simultaneously through virtual laboratory. Each of the machines has three extra un-configured virtual network interfaces. The students need to configure the machines to enable communication on the same isolated VLAN, while the isolated VLANs are bridged together, so that the machines connect to each other when configured correctly.

 Beside to substantial additions to NCSU VCL, we have also implemented web services, integrating VCL with our LMS of choice (Learning Management System – Moodle), University identity management system, assessment framework, teleconferencing facility and two social networks (Twitter and Facebook). Exposing system functionality in the form of service enhances our “Laboratory as a Service” paradigm. The services offered by our virtual laboratory implementation are described in the following section.

IV. Laboratory as a Service

 Laboratory as a service (LaaS) paradigm is an analogy to the contemporary buzzwords like SaaS “software as a service”, IaaS “infrastructure as a service” etc. in a way that the laboratory users do not need to establish an expensive laboratory infrastructure (e.g. physical laboratory) themselves, to buy or develop software needed and to manage the system themselves. Instead, they remotely access the infrastructure that is established and managed by others and offered as a service to interested parties – in our case to universities, students and teachers.

 In our case virtual computing laboratory is offered as a service. It is a complex system consisting of software and hardware parts. The software part includes a web portal, through which basic management of virtual machines is possible. However it lacks the capability of straightforward integration with learning management systems in a way that would enable users (students as well as teachers) access the basic functions of the virtual laboratory from a known environment. These functions are among others assignments definition, grading and authentication for teachers; reservation, startup and release of virtual machines needed for assignment completion for students. In this section we describe the way to use virtual laboratory as a service offered to the learning management system. Moreover, typical scenarios of use by students and teachers are given.

 Our implementation extends the functionality of learning management system Moodle 1.9 in a way that enables access to VCL, and VCL functionality is exposed as a service to its users – students and teachers. The laboratory service is accessible at any time of day and from everywhere, enabling time and space shift: it is no longer necessary that students and teachers meet in a physical laboratory within the predefined hours. There are several reasons why Moodle was chosen from a wide choice of learning management systems available: it is open-source, rich in functionality, reliable, free, widely used and regularly updated.

 Our integration service is visible within Moodle as a new type of activity and is straightforward to use within existing courses just like any other activity. When creating new VCL assignment, a teacher needs to define assignment name, write assignment instructions and choose one of the existing virtual machine images within the virtual laboratory. If new image is needed, it needs to be created separately and imported into VCL beforehand. A teacher may also define the time limit for assignment completion, assessment service (if applicable), deadlines, etc. After that, the activity is available to students. If an assessment service is selected, the service will grade the work accomplished in the VCL and the grades will be returned and saved in Moodle. Without automatic grading, the manual assessment needs to be performed before the virtual machine is released, since all changes are lost afterwards. A possible solution is a parallel activity, enabling a student to upload a certain file into Moodle before the assignment times out, while the teacher may grade the work later.

 When a student chooses the VCL activity in Moodle, he or she can see a list of all valid reservations and previous tries. When making a new reservation, he or she chooses between immediate start and a reservation of resources for later time. If any of the reservations is due, the student may enter the virtual laboratory; however if he doesn’t do it in 15 minutes, the reservation is cancelled. When the assignment is graded automatically, the grading service is to be activated by the student before the reservation is released. The typical scenarios of service use available to teachers and students are graphically reviewed in Figure 3.

Figure 3 Virtual laboratory is available to students and teachers as a service, accessible from LMS.

V. Results and Discussion

 In the spring semester of 2010, our virtual laboratory was used by more than 300 students without significant technical difficulties. In this section we present some interesting statistic on the system load, simultaneous reservations and performance.

 The majority of students were allowed two weeks to accomplish the observed assignments, while a smaller group was allowed an extra week. Distinct assignments were prepared: two of them requiring only one virtual machine and each of them assigned to one half of the students, while the third required three virtual machines and was assigned to all of the students. If the two assignments were completed each in the first try, a total of four reservations were needed, however the students were allowed to enter the virtual laboratory, make any number of reservations and use the virtual machines at will. The reservations were set to 60 minutes, and the students were allowed to make extensions of up to 60 minutes of additional duration.

 A total of 317 students have made at least one reservation, while the total number of virtual machine (VM) reservations in the observed 20 days was 2840. Note that the third assignment takes three virtual machines and the actual number of assignment reservations was 1347. In the rest of this section, when not explicitly stated otherwise, with a term reservation we refer to the virtual machine reservation. So, in order to accomplish two assignments, the students needed four attempts on overage. However the reservations with the duration of less than 25 minutes cannot be considered as serious attempts, more likely the students were just trying the system.

 Students spent a total of 934 hours in virtual laboratory, of those 550 hours in the third assignment. The average reservation duration was 44 minutes and 24 seconds; more detailed frequency diagram is shown in Table 1. The reservations without duration (0 minutes) are either that made for future time and later cancelled or not realized, or that interrupted while preparing the image or any time before the login process was finished. The majority of students were able to complete the assignments within 75 minutes, while the longest reservation lasted for 120 minutes.

Table 1 Frequency distribution of reservation duration

 We were also logging the circumstances of reservation endings. Most of the reservations (806) were ended by user request. The next 320 reservations were ended after the allotted time passed and were not prolonged (the students were able to prolong the reservation if they couldn’t finish on time). Only three times the reservation was not possible since the equipment was unavailable (the virtual laboratory was full). 130 of the assignment reservations were for future time, but 59 of those were not realized – the user didn’t log in. Probably the students have made the reservation for future but nevertheless tried to enter virtual laboratory immediately – and succeeded. So the future reservation was not needed anymore, however they didn’t bother to release it. In 73 cases, the user disconnected from the virtual machine without releasing it and did not reconnect later. Beside intentional disconnection, sometimes, when configuring network interfaces, the students by mistake disabled their own access to the virtual machine(s). They needed to begin from the start with new reservation and fresh images, however the old images were kept running until the timeout.

 Figure 4 shows the frequency break-up of the number of virtual machine reservations made by the same student. For example, 112 students have made less than 6 reservations each, while the total number of such students’ reservations is over 400. The majority of students needed less than 10 reservations to accomplish the assignments – beside 112 students mentioned above, Further, 116 students needed between 6 and 10 reservations. 19 students made more than 20 reservations – either from curiosity or because their sessions kept interrupting due to connectivity problems. In other words, while less than half of the reservations (48.9%) were made by 72.15% of the students with an average of 5.9 reservations per student, the remaining one quarter (27.5%) of the students, having more than 10 reservations each (16.8 on average), is responsible for the remaining half (52.1%) of the reservations. Further research is needed to evaluate why they used such a high number of virtual machines. If the reasons are technical (faulty connections, sessions keep disconnecting, etc), we need to fix the system. But if the reason is curiosity and trying new things, we have reached our goal - encourage students to constructive study and active decisions!

Figure 4 Number of reservations belonging to the same student

 Since virtual laboratories enable “time and space shift” – completion of assignments at any time of the day – we were also interesting in the time of the day and the dynamics of when the students most often access the virtual laboratory. As seen in Figure 5, most often the students accessed the laboratory late in the afternoon or in the evening, while the teachers usually prefer the morning and early afternoon hours. We have to point out that the observed 20-days period was in June, when all the lectures had already ended and the students did not have other obligations besides studying for the exams. So there was no reason why they could not access the virtual laboratory earlier. The least popular time was the hour from 6 to 7 in the morning - not even a single reservation was made in the whole 20-day period, while in the most popular hour from 22 to 23 PM the total number of reservations was 242.

Figure 5 Total number of reservations started by hour of day within the whole 3 weeks period.

 Figure 6 shows the actual number of virtual machines started by the hour. The maximum of 100 virtual machines in one hour was reached within the last hour before the assignment submission deadline for the majority of students. The second peak at the end (after three weeks) represents deadline for those students that were given an extra week. However, most of the time (in 274 out of 479 hours), especially in the first week, no new reservations were made. This shows that a lot of students postpone their duties although they have been warned that the system might be busy close to the deadline. Obviously, it remains an open question how to further encourage students to use the virtual laboratory in a timely manner and thus have better chance to improve the collaborative and constructive aspect of learning. That is, the students have to be encouraged to use the virtual laboratory not only as an environment where they perform their obligatory assignments, but also as a (virtual) playground where they try new things in a controlled environment and get in touch with more complex technologies without fear that they could break or spoil something important.

Figure 6 The number of new reservations, starting within an hour – the whole 20-day period is on x-axis.

VI. Conclusions

 In the paper we presented a practical and theoretical motivation to introduce a blended learning environment in a Computer Science courses at the University of Ljubljana. Although introduction of such environment does substantially increase the availability of laboratory environment for students, it turns out that by careful planning and design we did not introduce additional infrastructural requirements but merely shift from the classrooms to computer infrastructure in a cloud.

 Further, the paper presents our design of the virtual laboratory architecture and implementation with a standard service interface. We further gave the preliminary results of use of the virtual laboratory in a Computer Networking course. By now we used virtual laboratory also in Operating Systems course successfully, while in the next year we will expand its use to Algorithms and Data Structures and Network Protocols and Security courses. We are looking forward to expansion of our system infrastructure, which would enable support for more and more courses.

 Furthermore, the SAKE architecture is under consideration to be used in professional training courses for network equipment by the company NIL Data Communications Ltd.

Acknowledgement

 The authors wish to thank all the members of project team for their input, as well as to the anonymous reviewers for their constructive comments.

 The study is partly financed by European Union – European Regional Development Fund.

 

Reference

1.Anisetti M, Bellandi V., Colombo A., Cremonini M., Damiani E., Frati F., Hounsou J. T., Rebeccani D. (2007). Learning Computer Networking on Open Paravirtual Laboratories. IEEE Transactions on Education, Vol. 50, No. 4.
2.Ben-Ari, M. (2001). Constructivism in Computer Science Education. Jl. of Computers in Mathematics and Science Teaching, 20 (1), 45-73.
3.Barros B., Read T., Felisa Verdejo M. (2008). Virtual Collaborative Experimentation: An Approach Combining Remote and Local Labs. IEEE Transactions on Education, Vol. 51, No. 2, pp. 242-250.
4.Bochicchio M.A., Longo A. (2009). Hands-On Remote Labs: Collaborative Web Laboratories as a Case Study for IT Engineering Classes. IEEE Transactions on Learning Technologies, Vol. 2, No. 4, pp. 320-330.
5.Cooper M., Ferreira J.M.M. (2009). Remote Laboratories Extending Access to Science and Engineering Curricular. IEEE Transactions on Learning Technologies, Vol. 2, No. 4.
6.Garrison, D.R., Vaughan, N.D. (2008). Blended Learning in Higher Education. San Francisc Jossey-Bass.
7.Gravier C., Fayolle J., Bayard B., Ates M., Lardon J. (2008). State of the Art About Remote Laboratories Paradigms – Foundations of Ongoing Mutations. iJOE vol. 4, No. 1.
8.Greenhow C., Robelia B., Hughes J.E. (2009). Learning, Teaching, and Scholarship in a Digital age: Web 2.0 and Classroom Research: What Path should We Take Now? Educational Researcher 2009, 38, 246.
9.Hardaway, D.; Hogan, M.J.; Mathieu, R.G. (2005). Outsourcing the University Computer Lab. Computer , vol. 38, no. 9, pp. 100-102.
10.Lapuh Bele, J.; Rugelj, J. (2007). Blended learning - an opportunity to take the best of both world. Int. j.: emerg. technol. learn., 2 (3), 71-75.
11.Lapuh Bele, J., Rugelj, J. (2007). Providing feedback in web-based learning. In: AUER, Michael E. (ed.). 10th International Conference, ICL 2007. ePortfolio and quality in e-learning. Wien: International Association of Online Engineering.
12.Li P. (2009). Exploring virtual environments in a decentralized lab. ACM SIGITE Research in IT, Vol. 6, No. 1 (Jan. 2009), 4-10.
13.Li P., Toderick L. W., Lunsford P. J. (2009a). Experiencing Virtual Computing Lab in Information Technology Education. Proceedings of the 10th ACM conference on SIG-information technology education, pp. 55-59.
14.Lowe D., Murray S., Lindsay E., Liu D. (2009). Evolving Remote Laboratory Architectures to Leverage Emerging Internet Technologies. IEEE Transactions on Learning Technologies, Vol.2, No. 4, pp. 289-294.
15.Murphy M. C., McClelland M. K. (2009). My Personal Computer Lab: Operating in the "Cloud". Information Systems Education Journal, 7 (93).
16.Nančovska Šerbec, I., Strnad, M., Rugelj, J. (2009). Students' attitude to active forms of e-learning. In: M. Činčin-Šain, (ed.). Proceedings Rijeka: MIPRO, 100-103
17.Powell, V.J.H. et al., (2007). VLabNet: A Virtual Laboratory Environment for Teachnig Networking and Data Communications. Proc. ISECON 2007. Available at http://proc.isecon.org/2007/2335/ISECON.2007.Powell.pdf.
18.Schaffer H.E., Averitt S.A, Hoit M.I., Peeler A., Sills E.D., Vouk M.A. (2009). NCSU's Virtual Computing Lab: A Cloud Computing Solution. Computer, vol. 42, no. 7, pp. 94-97.
19.Singh, H. (2003). Building Effective Blended Learning Programs. Education Technology, 43 (6), 51-54
20.Slanšek, S. (2009), E-learning: principles and implementations. University of Ljubljana, Faculty of Computer and Information Science.
21.Sluijsmans, D., Dochy, F., & Moerkerke, G. (1999). Creating a learning environment by using self-, peer- and co-assessment. Learning Environments Research 1, 293-319.
22.Vouk, M.A. (2008). Cloud Computing – Issues, Research and Implementations. Journal of Computing and Information Technology - CIT 16/4, 235–246.