Best Practices and Common Evaluation Criteria in Distance Education

The purpose of this study was twofold:

to look at various accepted standards for best practices in distance education and general guidelines for assessment
and to compare those standards with what is actually assessed on distance education program evaluation instruments.

Introduction & Methodology

For the first part, this study looked at accepted standards from the regional accrediting agencies, from professional school accrediting agencies, from the Council of Graduate Schools, from WICHE and NCHEMS, and from professional associations. Some of these documents have subsequently been superceded by the recent "Best Practices for Electronically Offered Degree & Certificate Programs" developed and approved by the eight regional accrediting commissions. The standards referred to in this latest document continue to support the criteria used in this study.

After reviewing the different best practices, it was clear that there was much similarity on the major categories that define best practice. Those categories are:

Organization & Planning, specifically with regard to governance structure and institutional support for the distance education program, but also related to assessing market demand and currency of content;
Teaching/Learning Transaction, or course-student related practices which are further broken down into course design practices, course/program content practices and program design practices;
Delivery Methods, which includes support for the differing technologies available at the faculty, staff and student levels;
Educational Support, which questions access to library resources as well as availability and coherence of supplementary course materials;
Student Support Services, which includes everything from registration processes and financial aid to academic advising and career counseling; and finally the capstone,
Evaluation & Assessment, to ensure that the program is meeting the needs of the students and the institution.

A diagnostic instrument was then created based on the synthesis of the most often mentioned standards for best practice. (Appendix)

The second part of the study used the diagnostic instrument to compare the best practice categories to sample evaluation instruments used by Continuing Education units offering distance education programs. Evaluation tools were requested of all the major distance education providers in UCEA. Twenty-one (21) institutions submitted their evaluation instruments for this study. The results presented here are based on those 21 institutions, with most of the largest distance education providers in credit bearing courses included.

Data were coded and entered in an excel worksheet. A score of 1 was given to each university whenever a criterion within each of the categories previously defined was mentioned in its evaluation instrument. If a criterion was not mentioned in a university’s evaluation instrument, a score of zero was then given. It is important to note at this point that deciding whether a university mentioned criteria within a category or not required interpretation of the concepts. This may be taken as a general deterrent to the credibility of the results of this study. The negative impact, however, was minimized by having three different individuals review the data and compare their results for trustworthiness.

Results

The results of this study are presented in three different sections. The first section (slides 1-14) explains and describes the methodology used to conduct the study. The description of the methodology includes the data collection process, the organization of the data within the delineated categories as well as the questions related to each category.

The second section presents the data analysis process. Data in this study are analyzed in two different ways. The first part (slides 15 to 26) uses bar graphs to report the percentage comparability of each of the 21 institutions with respect to the criteria included in the major standards of best practice identified earlier. It is important to stress that for most of the bar graphs in that part only the number of institutions who mentioned those criteria is reported.

For this reason, most of the percentages do not total 100. In the second part of this second section (slides 27 to 37), institutions are compared based on the number of criteria they mention in their evaluation instrument for each category. For that purpose and for each category, institutions are divided in to three groups: high (institutions mentioning more than 66% of the criteria within a given category), medium (33-65% of the criteria within a category), and low (less than 33% of the criteria within a category). Results in that part are presented in the form of pie charts based on the percentages of institutions that responded to the given category.

The third section (slides 33 to 37) presents general conclusions, suggestions and final comments based on the results of the study.

General Observations

In general, there was a higher response to questions related to the teaching/learning transaction of a specific course than there was to questions related to program coherence. As programmatic coherence is a major factor in accreditation reviews, it seems that some attention needs to be paid to evaluating course integration within the larger program rather than just content coherence within individual courses. It is, however, highly probable that institutions are conducting programmatic reviews of the distance education programs, but that those assessment instruments were not submitted for this study. Nonetheless, the lack of response to the program coherence question does indicate that we may not be asking the right questions of our students. The three institutions which had something on their evaluation instrument that could be related to program coherence, only asked "Does this course fit with your major course of study", or similar wording; no one specifically addressed the issue of program coherence.

The same is true for questions related to student support services. The highest response rates were for a generic student support question and one regarding the registration process. Very few institutions were asking questions about payment methods, financial aid, or career counseling opportunities, all of which are to be made available to students at a distance.

Another item of note, was the change over the past few years in the number of questions regarding the reliability of the delivery method. The highest percentage of response for delivery method questions came under the appropriateness of the delivery method not under its reliability. This may indicate that the technology has advanced to the point that now it is simply expected to work well all the time and the reliability factor is, therefore, not as pertinent today as it was six years ago when WICHE first developed the "Best Practices" guidelines.

The study also demonstrated that there is room for improvement on the way student evaluation instruments can be used to gather demographic related market research data. In fact gathering the right demographic information will allow institutions to better direct identify and inform their potential markets. In many cases, demographic information is limited to age, gender, and the like and is requested by a limited number of institutions (48%).

There was a fairly large disparity in the breadth of questions asked on the evaluation instruments of the institutions included in this study. A few appeared to have a more comprehensive approach to their assessment system. Those are: University of Maryland-Baltimore Campus, University of British Columbia, University of Phoenix, Washington State University, University of Wisconsin, and the University of Waterloo in Canada.

Conclusions

This study was undertaken by the Quality Assurance Community of Practice to help UCEA members refine their distance education assessment instruments. The categories outlined in the best practices documents should be the categories used to assess distance education courses and programs. At this point, there appear to be only a few institutions that have assessment tools that are based in best practice standards. It is our hope that this study will cause distance education program directors to take another look at their evaluation tools to ensure they are measuring what they need to. It is clear that the generic on-campus end-of-course evaluation instrument does not encompass the breadth of questions necessary for evaluating a distance education program.

As a final word, we would like to extend our appreciation to the institutions that participated in this study by submitting their evaluation instruments. We welcome your feedback on this study and on how we can improve it to help UCEA members meet assessment goals.

Krista Rodin

Abdou Ndoye