Supporting group learning and assessment through Internet

Liane Tarouco: UFRGS/PGIE, Av Paulo Gama 110, FACED sala 810, 90046-900 Porto Alegre – RS - BRAZIL
Antonio Rodrigo de Vit – UFRGS/PGCC – Campus Vale, Porto Alegre – RS - BRAZIL
Luciano Hack - UFRGS/PGCC – Campus Vale, Porto Alegre – RS - BRAZIL
Marlise Geller - UFRGS/PGIE, Av Paulo Gama 110, FACED sala 810, 90046-900 Porto Alegre – RS - BRAZIL

Abstract

Internet is being intensively used for distance education. Usual services like email, chat and videoconference are not enough to fully support for the proper handling of interactions without extensive non-automated work. To support group work related to learning activities it is necessary to develop applications to handle participants’ contributions and to consolidate contributions by providing summaries of the discussion. At the same time they must be integrated with group decision support tools to help students draw conclusions from what was discussed. Assessment in this kind of learning environment also lacks better support. This paper presents a learning environment developed to support group work and student assessment through Internet for distance education.

Keywords: group learning, decision support system, assessment, distance education 1. Introduction

Computer-mediated distance learning (CMDL) is an uncommonly bright star on the horizon of innovations in higher education. According to a recent study by CCA and Associates, 30 percent of colleges and universities have at least one distance learning program in place and an additional 28 percent are planning such programs. [Tucker]

Although Internet use in distance education grows exponential nowadays, services used in the beginning were only those having strong relation with ancient forms of distance education. Distance education started using post services and even in those days, many courses used on large scale Post Office distribution of printed material. Other mass communication media, such as television and radio or recorded tapes with video or audio where also very common. Some Internet services started to be used to replace or complement these kinds of service technologies used as shown in table 1.
 

Ancient forms of distance education Internet based environment
Printed text books HTML pages
Exercises Forms and CGI
TV broadcast  MBONE
VHS tapes Real Video, MPEG
Audio tapes Real audio
Assesment Forms and CGI 

The use of the Internet offers more possibilities of new forms of interaction, as shown in the following figure:

The use of the Internet for distance education resulted in many advantages:

To support more kinds of interactions derived of group work related to learning activities, it is necessary to develop applications to handle the participants’ contributions. Collaboration strategies are required to improve the quality of the information provided to support decisions, but this srtill remains na innovation in distance educations.

Despite all the bells and whistles of groupware, e-mail continues to be the most widely used groupware component. Hence one can count on all the automated work provided by email or news service servers to receive, store and/or properly forward messages. But no matter if only email or news service are used for group communication, as well as other forms of interaction, like chat combined with videoconference or with other multimedia environment (such as The Palace or Virtual Reality based environments), the results from group activity are huge amounts of text derived of participants’ contribution. When it comes to the consolidation phase of the work, it is usually neessary to have a lot of non-automated tasks to select the main ideas from the many ones presented almost randomly during the discussion. It is also necessary to provide a voting system to support group decision about the consolidated results from discussion (ordering, reordering, selecting, discarding etc).

This work describes the strategies and tools designed to accomplish automatic grouping of ideas (with focus on common themes) for the organization of contributions from steps in a group decision process as well as for student assessment in a framework derived from Kirkpatrick’s model. A set of tools was designed and developed to support distance education cooperative work. The following section describes the group tools, and the next describes the assessment ones.

2. Group tool

This work starts from the supposition that group activity results in a log file containing interchanged texts (synchronously or asynchronously) during a group debate. Such inputs are submitted to a set of routines that groups ideas automatically. The system uses a WWW-based interface and log files derived from the group interchange of ideas. The present results were built from previous works developed at UFRGS.

The first work considered to start the present project was a tool called "Issue Analyzer", which provides natural language recognition. It also contains an algorithm for calculation of similarity between contributions of participants of a group discussion. This prototype was implemented on Sun stations and used the software RPC (Remote Procedure Protocol) for intercommunication between a server and client applications. It provided the grouping of similar ideas generated during the brainstorming phase and organized them according to definitions of the group.

Another work previously developed, named Eurekha, showed the importance of information retrieval techniques to support the organization of textual information. It pointed to the usefulness of this kind of support in an environment with information overload like Internet. This solution uses methods for grouping textual objects. Objects are organized automatically in similar object groups, facilitating their location, manipulation and analysis.

Both systems are directed to natural language (in written form) processing. In natural language processing one of the most important aspects is the formal language representation. This work applied the Discourse Representation Theory. This kind of system faces big challenges: linguistic aspects of natural language representation and handling of problems like ambiguity, references to names, pronouns, etc. As a result, this solution also included the grammatical classification of sentences.

The present work aimed at finding out a tool for group decision support system that is able to handle natural language input.

2.1. The Model Proposed

The system designed used the results of previous works mentioned above. It starts with a Module I, which handles log files (derived from chat sessions using softwares like CuSeeMe, The Palace and similar ones). Files are split into smaller units (files with only one sentence or contribution). Those smaller units are submitted to a routine that performs similarity calculation. This module processes each sentence, which results in a properly labeled version according with a Syntactic Derivation Tree (ADS).

Module II (previous work named Eurekha) handles sentences by grouping them in clusters according to similarity. In this step the pre-processing of natural language is done. It is considers that the documents have already been corrected through an orthographic checker (task also performed by module I). Besides, it is advisable that the texts suffer some normalization of terms. However, these procedures are not conditionings to the operation.

The aditional modules (III to V) were built during the development of this project (Modules III to V). Module III handles the resulting cluster of sentences and identifies the most frequent/relevant words that to be used in the creation of a set that represents each cluster.

Module IV indexes words and links them to original sentences for further information retrieval, in case one wants to know which sentences provided more words representative for clusters (and of course who made those contributions).

Each group of words becomes an alternative in a voting process. The voting results in a hierarchy of representative ideas set by the group. Module V implements this voting process. An HTML page with the results of this stage of the discussion is presented to the participants of the group to feed one more round of discussion.

2.2 Extended results

A prototype version of the system is running. It provides all the funcions described above:

The results of this process may be used as a starting point for a new step of the group work where new discussions will use the results from previous consolidated work. This cycle can be repeated endlessly.

The tool will be used not only to support drawing conclusions from consolidated sentences, but, as it also keeps the original sentences and their creators, it is possible to assess the productivity of each participant. Not only the amount of contributions can be counted, but it is also possible to identify contributions that originated consolidated clusters selected by participants as the most relevant and representative of the discussion conclusions. So it is possible to identify those students whose contributions are more significant, since they are included in the clusters most accepted by the group.

3. Tools for assessment in distance education

Learning assessment has become a challenge to those working with distance education. In their definition of learning assessment, Bloom, Hastings and Madaus, propose that the evaluation process should include a great variety of evidences that go beyond the traditional final examination pencil and paper based. In a face-to-face context, instructors use more than just formal mechanisms to evaluate students. Body language, participation and the quality of questions proposed by students are good indicators of learning.

Assessment in distance education, however, is usually restricted to formal mechanisms.This happens not only because of the lack of accessible mechanisms to assist instructors in this task, but also, because many professors prefer to work with more traditional methods of evaluation.

A research conducted by Dirks [Dirks] with instructors of distance learning MBA programs presented some questions like:

Considering all the time you spend on this course (100%), how much of that time do you think you spend on student assessment (writing tests, grading, giving feedback, and reporting scores)? The responses indicated that the less experienced teachers, 2-9 years teaching, spent over half their time on assessment (56%), those with 10-19 years experience spent less time on assessment (46%), and the career instructors, 20+ years teaching, spent the least amount of time doing assessment (35%).

The primary reasons given for having assessments include providing feedback, giving grades, and motivation. The four types of assessment most commonly mentioned are: case studies, exams, papers, and projects.

All of them require extensive time to grade and are very difficult to get automated help to do the evaluation.

Other researches also point out that common means of assessment in distance education are:

Learning theories state that group learning has significant relevance and must be supported also in distance education. Participation in group activities, cooperation and collaboration must be supported and graded. But there is lack of good tools to help evaluate the participation of distance education students in group activities.

However, thanks to the evolution of networks and computers and specially the Internet, mechanisms that can fill this gap have been created litlle by little. New tools were provided to keep track of students’ activities and interactions with the learning environment, with colleagues and with the professor. This kind of information presents new opportunities for monitoring the way students learn and for learning assessment.

As stated by Tucker, the CMDL (Computer Mediated Distance Learning) classroom is usually a far more scrutable educational environment than the physical classroom. The virtual classroom's electronic data storage, retrieval and exchange system (i.e., the text of student and faculty transactions, communication logs, file structures and information presentation algorithms that exist on the file server's hard disk drives) represent concentrated, structured and highly accessible artifacts of the learning transactions. These artifacts can be retrieved, analyzed and reported via highly economical, automatic processes operating in the background of the communication system. [Tucker]

Next, we present and describe the implementation process of a solution proposed for assessment in distance education mediated by Internet that applies innovative approaches based on Kirkpatrick’s model to training evaluation.

3.1. Assessment tools and Kirkpatrick’s model

There are today some systems available in the market that provide detailed information on the evolution of the student working at distance mediated by Internet. Examples of this kind of system include CyberQ, WebCT and AulaNet. Statistics of usage and regular approaches for testing, such as multiple choice quizzes and term papers, are usually available in this kind of systems.

Donald Kirkpatrick, [Kirkpatrick] developed a model of evaluation, used in training program, which recommends a division of the evaluation process in four levels:

Evaluation at the reaction level measures how the participants of the program react to it. It may be called an estimate of students’ satisfaction.

Learning can be defined as the extent to which participants change attitudes, improve knowledge, and/or increase skills as a result of attending the program.

Behavior can be defined as the extent to which changes in behavior occur because the participant attended the training program.

Results can be defined as the final results of the students’ participation in the program.

The research developed at UFRGS aimed at designing a set of tools to complement regular ones used to assess distance education students. The following mechanisms have been selected to assist in the evaluation process in this framework:

It is important to stress that this work proposed is not an alternative approach to assessment, but a complementary one, meant to be combined with others.

Each one of these tools is meant to help evaluation at different levels in Kirkpatrick model.

For the implementation of the tracking tool, a generic interface was designed to mediate all the interactions of the students with the server. Interactions are processed in real time (for authentication purposes) and periodically in batch to generate records in a student database.

To implement the voting, consensus and self-evaluating tools, a set of CGI programs was developed to allow a dynamic creation of pages with questions and propositions (HTML forms) that are presented to students, which originates answers to be collect. [Tarouco].

The following diagram shows the evaluation sub-system. The design of this system aimed at creating an easy self-learning system, so that the faculty or teacher using it would not need extensive computer knowledge. The whole system may be operated through basic browser-handling skills.

A mailing system included in the package allows the interchange of messages between students and instructors as well as among themselves. Email boxes are handled by each participant for asynchronous communication. A chat room server gives support for synchronous communication.

This is a system aimed at supporting formative evaluation, so it not only allows the tracking of all communication and students’ activities and progress, but it also provides the tool to evaluate reactions, according to Kirkpatrick's model. It is also meant to support collaborative learning,  where students work together in groups to accomplish a common learning goal and have to make group decisions from a distance. The consensus and voting tools were designed to facilitate dynamic building of questions that will be answered by participants to express they opinions.

The navigation routine intercepts all communication and, by doing that, allows its tracking. All information is filed in a database, so that the instructor may retrieve students’ profiles at any time.

  4. Conclusion

To assess face-to-face learning, instructors use other tools than the formal ones (quizzes, term papers, etc). Questions presented by students and even body language provide feedback especially in level 1 (reaction), according to Kirkpatrick's model. In distance education it is necessary to have complementary tools to get feedback from reactions of students to stimuli proposed as learning activities. The complementary tools proposed in this work include tracking of all interactions among students and other entities mediated by the system, as well as flow control, voting and consensus.

This work also describes a tool able to inspect group contributions, consolidate and present them again to the group for selection and ordering of topics resulting from  the clustering process.

The tool proposed in section 2 is being used not only to support drawing conclusions from consolidated sentences, but, as it also keeps the original sentences and their creators, it is possible to draw conclusions on the productivity of each participant. So it is also a complementary assessment tool.

All the tools described in this work were designed to support group activity in distance learning as well as assessment of this kind of activity, because we believe that cooperation and collaboration are key components in distance education and that this kind of approach should be promoted. Both are running and being used in distance learning at Universidade Federal do Rio Grande do Sul (Federal University of Rio Grande do Sul).
 

5. References

[1] T.X.Bui. Co-op a group decision support system for cooperative multiple criteria group decision
making. Germany: Springer-Verlag, 1987. 250p. (Lecture Notes in Computer Cience).

[2] M.Dirks. How is Assessment Being Done in Distance Learning? 1998.
http://star.ucc.nau.edu/~nauweb98/papers/dirks.html.

[3] D.L.Kirkpatrick. Evaluating Training Programs - The Four Levels. 1998. Berret-Koehler Publisher, Inc -
San Francisco.

[4] Tarouco, Liane. Pereira Luiz. Quiz Building on the Web without CGI or Java Programming. Interactive Multimedia 97. SALT. Washington-DC, 18-22 de agosto de 1997.

[5] R.W.Tucker. Assessing the Virtual Classrooms: A Progress Report.http://www.intered.com/edv5n2.htm

Authors’ biographies

Dr. Liane Tarouco is a professor at Universidade Federal do Rio Grande do Sul, Porto Alegre, RS, Brazil. She is chairperson of the Graduate Program "Informatics on Education", where she runs research on teleducation and computer networks. Served as representative of Brazil at IFIP-TC6 (International Federation of Information Processing) and is today member of coordination committee of Brazilian Academic Network (RNP-Rede Brasileira de Pesquisa). Published books include "Data Communication Networks" (in portuguese) by LTC-Livros Técnicos e Científicos Ltda and "Computer Networks: LAn and WAN" (in portuguese) by McGrawHill.

Marlise Geller is a student at the Graduate Program "Informatics on Education" (PhD level). She is also a professor at ULBRA (Universidade Luterana do Brasil), Canoas, RS, Brazil.

Antonio Rodrigo de Vit is a graduate student at the Computer Science program at UFRGS (MSc level). He is also a professor at UNIJUI (Universidade de Ijuí), Ijuí, RS, Brazil.

Luciano Hack is a graduate student at the Computer Science program at UFRGS (MSc level).  He is also a professor at UDESC (Universidade do Estado de Santa Catarina), Florianópolis, SC , Brazil.