The Rochester Institute of Technology’s Online Learning group manages the campus-wide course management system (CMS) now used voluntarily by over 60 percent of the faculty in 54 percent of course offerings. The Online Learning group has been data-mining information from the CMS for over five years to better understand what features are being used, which colleges and departments are using it the most, and what level of activities on key features are in each course.

The RIT “Terms of Use” policy linked on the CMS and all the Online Learning Web pages gives the Online Learning group considerable authority to explore this data:

You grant us the right to use, reproduce, display, perform, adapt, modify, and distribute in any form, media, or method now known or hereafter devised, for any purpose, any material you submit or post through any interactive feature of our Site, including message services or bulletin boards.

The “Panopticon” effect of course management systems represents a threat to faculty that we have avoided at RIT. The Online Learning department will allow department chairs into an online course if they request it, but in those rare instances the faculty are notified. There is a legitimate need for administrators to observe, mentor, and evaluate faculty and there is no better way to do this in online courses than to join the class. This is similar to observing campus classes, but surprisingly it is seldom requested. Most of the requests we receive from department chairs are associated with faculty illnesses or absences and the department steps in to provide support. No reporting is generated from the course management system data for faculty assessment, comparisons, or other punitive or restrictive purpose. David Noble brought these risks to light in 1997 with a series of articles on “digital diploma mills.” Although other institutions may monitor faculty time and activities online, RIT Online Learning remains focused on identifying aggregate trends in feature usage and effective teaching practices.

Although course management systems capture a wealth of information about individual users and courses, none of the four systems that RIT Online Learning has used in the last 10 years (FirstClass, Blackboard, Prometheus, Desire2Learn) provided reports that were useful to better understand and identify effective online teaching and learning practices. For almost five years now, RIT Online Learning has generated detailed reports of our own to help identify factors that might be associated with effective teaching. At the heart of this reporting is a record for each class of the number of messages posted by the instructor, the number of messages posted by students, the number of students, the number of news announcements posted by the instructor, the number of notes left by the instructor in the online grade book, the number of notes left by the instructor in the online drop box, as well as counts of all the other features used in a particular course. Once a quarter our system administrator runs a series of scripts against the CMS databases and provides an Excel spreadsheet to one of the Instructional Technologists who analyzes the data. This data is extracted from the CMS databases after the fifth week of classes so that summaries can be developed and disseminated during the current quarter.

The key metric used in the most influential reporting of this data is course “interaction.” A total of all the online discussion postings, course news announcements, and notes left by the instructor in the grade book and drop box is compiled and then divided by the number of students, plus one for the instructor. This metric is used to rank the courses, and the top 50 courses each quarter are recognized as “Heavy Hitters.” The top 50 faculty members are recognized in a news message broadcast through the course management system (currently used by over 60 percent of the faculty). The college with the most active courses overall is also recognized in an email to the dean from Joeann Humbert, director of Online Learning, and the most interactive courses in that college are identified and the faculty are included in the distribution. This “low-stakes” faculty “award” helps recognize those faculty who often work very hard to promote student and faculty interaction. Although fully online courses often dominate the top 50 list, many more campus courses, usually blended or hybrid, are making the list as well.

Why do we emphasize interaction as opposed to other potential indicators of quality courses? Our instincts, a fundamental belief in constructivist learning theory, our faith in reform documents like the AAHE “Seven Principles for Good Practice in Undergraduate Education” by Chickering and Gamson, and, more recently, our interest in the “Community of Inquiry” model developed by Garrison and Anderson, tell us that this simple metric is a reasonable indicator of instructional quality and that no other single metric can be so easily derived. A research project we started last year confirms our notion that this interaction metric is a good indicator of effective instruction in our fully online courses. In fall 2007, all the fully online courses were ranked by this metric, student surveys were administered, and the survey results were summarized into four quartiles ranging from the “most interactive” to the “least interactive” sets of courses. Here are the statistics on the percentages of students in each quartile who agreed or strongly agreed with important statements about course effectiveness:

Legend: Q1 = most interactive quartile of online courses, Q4 = least interactive quartile of online courses

Because of the presence of the National Technical Institute for the Deaf at RIT, a small but significant number of deaf and hard-of-hearing students are enrolled in the fully online courses in this study. The survey data indicates that these students, as well as our ESL students, have even an stronger perception of how the quantity of interaction affects their course experience.

Our Institute Review Board reviewed and approved the research project for this year. Although other institutions have developed metrics that focus on faculty performance, RIT Online Learning has focused on research and faculty recognition to promote continuous improvement in our courses. The research project will continue for two more quarters and then the responses will be aggregated. We have a special interest in understanding how online interaction can benefit our deaf and hard-of-hearing students, but the overall results will help to inform us on creating new faculty development initiatives.

Joeann Humbert is director of online learning and Richard Fasse is the instructional technology specialist for the Educational Technology Center at Rochester Institute of Technology.

This article first appeared in Magna Publications' newsletter Academic Leader. If you are an academic dean, provost, academic vice-president, department chair/head or have any role in academic leadership, then Academic Leader is for you! But don't take our word for it--try it free for 3 months and make the decision for yourself!