The Pendulum Swing of Standards and Evidence

May 1, 1998

Vol. 1, No. 5, May 1998

by Larry A. Braskamp and David C. Braskamp

Colleges and universities are now being conceptualized as learning communities- communities in which the most important persons are the students as learners. Faculty and the surrounding learning environment-the curriculum, social activities, facilities-exist to foster the social, physical, spiritual and the intellectual development of students. In the 1990's, the Goals 2000: Educate America Act, the Education Commission of the States and the Association of American Colleges and Universities have all called for higher education to reform itself, putting student learning first. Barr and Tagg (Change, 1995), among many others, label this new way of thinking a paradigm shift, whereby a college no longer is to be viewed as a place to provide instruction, but as one that exists to produce learning (p. 13).

The standards movement has also centered around the use of student learning and development as the main indicator of quality. Standards have become associated with what students are able to do and know at a given time in their education. The move toward the centrality of student learning has been beneficial in many ways. When standards are connected to student performance, they provide a very compelling argument for refocusing the definition of quality in higher education. By linking standards and performance, student learning and development becomes the starting point for examining program quality (i.e. program and institutional effectiveness). Moreover, by focusing on a few indicators of quality (e.g. pass rates on professional licensure exam), educational leaders can encourage the faculty to meet clear measurable goals. And a set of a few measures to reflect quality are also easily understood by the significant stakeholders, e.g. parents, politicians.

Standards and evidence have always played a prominent role in accreditation, since they are fundamental to the evaluation of quality. In accreditation, the change has not been so much a paradigm shift as a pendulum swing. Over the past decade, the accrediting community has begun to redefine academic quality, moving from inputs and resources (facilities, student excellence at entry into college) to process (quality of the collegiate experience) to quality that is based on student learning "outcomes." In short, we are now in the midst of a pendulum swing from "inputs" to "outputs,"although not everyone agrees that accreditation should ignore curriculum, planning, and support services.

What does this emphasis on the assessment of student learning and development mean for accreditation? What issues emerge with this new emphasis? What new conflicts and challenges do we face? How can assessment of student learning both meet the needs of the accreditation and learning communities? What is the role of accreditation when a learning community perspective is stressed? In this brief paper, we offer some challenges and suggestions.

Select key terms. The first challenge is definitional. Outcomes can refer to any result or product. It is a vague and inclusive term, that would be best deleted from our discussion of standards and evidence. It does not reflect a developmental perspective; it gives the image of a factory model in which something is produced rather than developed. Life long learning and development seem to be out of place in such an image. It also has an implicit dehumanizing connotation. (The same argument can be made for input.) We prefer terms such as student learning and development, or student performance.

Quality is another inclusive word that need redefining. Quality needs to be defined in terms of criteria and standards. Criteria provide the bases for judging quality. They are the reasons for someone to conclude that an academic program is judged to have quality. For example, a program can be considered of high quality because it enrolls excellent students, the faculty are competent, or the graduating students all pass a professional exam. Standards are similar to expectations. They represent a desired level of performance within a particular criterion. In one sense they are guides and goals for everyone to strive for. For example, they can help specify what and how to determine if the students are achieving at the level they are expected to learn or perform.

Our challenge is then to develop a language that invites the relevant audiences-presidents and faculty from all disciplines-and engages them in a dialogue that focuses on the development of students within a learning community environment.

Insist on high standards without standardization. The second challenge is to have high expectations without undue rigidity, and to be demanding without becoming overly prescriptive. Performance standards of student learning can unfortunately become a vehicle for standardization of expectations, goals, curriculum, and student performance. If so, homogeneity of actual and desired student learning may be the unintended consequence. Standardization can threaten creativity, assessment of multiple intelligences, and the promotion of individuality.

Our challenge then is to avoid the notion that high standards exist only through standardization, and to resist a dependence on compliance to predetermined quantitative benchmarks as the strategy for demonstrating academic excellence. We should not confuse quality with conformity to some centralized notion of quality. We must reinforce high expectations tied to student performance without resorting to undue standardization. In accreditation, it means defending the time honored principle of placing the institutional and/or program mission as the overriding guide in the assessment of academic quality. For example, requiring all students from all institutions to meet the same set of standards is viewed as too restrictive in our system of higher education. Standards supporting diversity and uniqueness must remain the hallmark of accreditation and of higher education.

Use a variety of evidence. Standards and criteria without evidence have little or no practical meaning. We must include evidence in our discussion of quality. The battleground then is really about what evidence can be used to define and demonstrate quality, which is our third challenge. Evidence is data: facts, information, descriptions, and statistical summaries that are used to determine if the standards of quality are met. It represents the results of measurement, assessment, testing, counting, observation, and analyses. The selection of evidence about student performance is of two types. One type includes quantitative records such as graduation rate, alumni and employer ratings, and percent of students having employment at graduation. The second type includes paper and pencil achievement tests, authentic assessment such as portfolios, performance tasks, complex assessment strategies including capstone courses, and a variety of measures and observations of skills, competencies, attitudes, and values. The former represents indirect proxy measures of student learning and development whereas the latter are viewed as direct indicators of student performance.

The challenge is to think creatively about a wide and varied set of student performance measures. In selecting assessment measures, we must recognize a fundamental paradox: the more we concentrate on a single assessment strategy (test or indicator of quality) to improve the quality of what is being assessed the more likely overall quality will decrease rather than increase. For example, if we focus too much on using students ratings of teaching to measure the quality of teaching, the quality of teaching in the aggregate on a campus will arguably be less effective. Faculty will more likely become less diverse in their teaching styles, less sensitive to individual student needs, and less likely to try innovative teaching strategies. Students often give instructors less favorable ratings when the instructors try unique pedagogical strategies. We must be careful that the pendulum does not swing too far, so that we try to search for the one "perfect" measure. Multiple measures that result in a collage of student performance is more credible than one measure or indicator of student performance, because it better demonstrates the diversity of standards of quality.

Focus on academic quality. The fourth challenge is to focus on the quality of the academic programs rather than on institutional or program administrative characteristics. That is, what evidence best portrays student learning and development? Which of the following are indicators of campus policies on student enrollment rather than measures of learning-default rate, persistence, professional licensure exam results, employer ratings, and professor's judgments of work done for credit in a class. Currently accrediting associations are required by the US Department of Education to include specific indicators like graduation rate in their assessment of institutional quality. Asking the accrediting associations to include them is reasonable, but the federal government also specifying the level of proficiency is going too far, e.g. requiring that 70% of all the students must graduate in order for the institution to be eligible for Title IV funds. These measures may not reflect student learning. Having 50, 79 or 100 per cent of the students graduating does not reveal very much about what the students have learned. This evidence reveals more about the institution's policies than student learning.

We must also be concerned about who is responsible for setting standards and collecting the evidence. In America, the federal government historically has not set standards and regulated them. Student performance measures make it easier for those outside the academy to influence the type of education offered to the students. While higher education needs to be responsive to society it should not give in and give up its perspective on what students should learn and how well they should learn it. In short, we prefer that faculty continue to play a major role in setting standards of academic quality.

Assess the entire spectrum of student learning and development. A fifth challenge is to ensure that quality is defined using the entire spectrum of student learning and development, including character, student experiences and the learning environment. If only measures of student performance at graduation are employed, resources and student experiences during enrollment may be viewed as unnecessary and even irrelevant, i.e. the learning environment and student experiences are not considered a part of the quality equation. This argument of course makes sense if we assume a perfect correlation between the educational environment (e.g., resources, facilities, faculty, student experiences) and student performance. In most accreditations, an imperfect linkage between resources broadly defined and student learning and development is assumed. As a matter of fact, the history of accreditation has been one of stressing resources and student learning experiences in defining and gathering evidence of quality.

The challenge is then to conceptualize an institution as a learning community, in which evidence is continuously being gathered, analyzed, and used by those responsible for the learning. That is, the faculty, those closest to the learning process, are actively engaged in the assessment of student development. Accreditation then can serve as the external audit of the internal monitoring and evaluation that goes on within an institution or academic program. It can be invaluable in insisting that each institution or local academic program be concerned with the connection and links between what students do with their time and the result of their engagement.

Differentiate quality based on student progress. A sixth challenge is to have a clear understanding of the use of student performance measures in determining who should attend, remain, and graduate from an academic program or institution. Evidence per se is neutral; it is valid only if it is used appropriately. We need different types of evidence for admitting students, monitoring the academic progress of students, and certifying students based on their successful completion of an academic program.

If quality is defined as minimal levels of skills and competencies of students at entry into college, it assumes that students need a specified level of competencies at entry into college before they can take advantage of the learning opportunities in college. At the extreme, colleges should stop offering remedial education and only admit "qualified" students, usually determined by some standardized test. However establishing minimum standards of performance for entry into college will greatly affect access and will impact various groups differently. Since socioeconomic status is correlated with academic achievement, a public policy that emphasizes minimal standards for entry into college will disproportionately affect the poor and those from under represented minorities. While the argument is that standardized tests favor meritocracy, it also favors a special blend of aristocracy given the high correlation between socioeconomic status and achievement. Since not all students can be above average, we can not afford to have an operational definition of quality that discourages enrolling the more challenging students.

Measuring the progress of student performance is particularly useful in guiding the faculty in academic quality improvement. It also allows an institution to establish its own unique agenda. Recently, many researchers in higher education have argued that quality be defined in terms of "value added", i.e., how much do students increase their skills, knowledge, and personal, social, physical, and spiritual development due to their enrollment in an academic program? To what extent has their collegiate experience enhanced their capacity to have productive careers and lead meaningful lives in a democratic society? Quality is to be viewed in terms of the institution or academic program's impact on the students. The measurement of quality thus means recording the progress in learning and development at various points in a student's collegiate experience. By using a value-added perspective, institutions can be very effective without having to enroll the most qualified students.

But satisfactory progress does not completely satisfy public safety and accountability demands and requirements. For example, knowing that a "pilot" has made significant progress in knowing how to get a plane in the air but lacks the skill to safely land it, gives potential passengers little comfort. A major challenge thus becomes one of setting minimal standards for entry into a profession.

Setting standards and selecting evidence for graduation and entry into a profession should be based on an open debate. Since assessment for entry into a profession or graduation is "high stakes", the dialogue should occur with the policy implications clearly articulated. For example, the debate should include the issue of how much does a professional need to know at entry into a profession and how can one develop most effectively as a professional. In short, the timing of measuring student achievement influences the definition of quality.

Distinguish quality based on student performance and institutional effectiveness. A seventh challenge is to make certain that the multiple missions of the institution remain at the core of accreditation. At most institutions, some combination of teaching, research, and public service (also known as outreach, engagement) are supported. If student performance becomes too much the focus of defining quality in accreditation, the public good of the institution becomes redefined in terms of the private gain of the students. Higher education makes multiple contributions to our society, and they must be recognized in any evaluation of quality and integrity. Practically, a multiple perspectives strategy employing multiple indicators and measures of quality and effectiveness is required.

Involve faculty in judging quality. A final-and perhaps the most important-challenge is to address the issue of the role of faculty in judging quality. If the pendulum swings too far, it will threaten the role of faculty as both self and peer judges of quality. If student performance is the only indicator of quality, faculty responsibility and authority in determining quality are lessened. If standards of quality can be established by only those outside the academy, faculty then become a minor player in determining quality. The external perspective of quality brings in the customer, an important player to be sure, but the customer is only essential and not sufficient. An over emphasis on easily measurable student indicators of quality can result in an unbalanced influence by the customer. For example, if we define quality by the percent of students having jobs at graduation, the perspective of the faculty on what students should learn for their lives as well as their multiple careers may not be adequately considered. Faculty also have the obligation to communicate to others the value of the academy reinvesting its core commodity-the pursuit of knowledge, creativity, and critical analyses. Historically in accreditation, faculty and the academic community have been the major instruments in judging institutional and program quality and integrity. Faculty must remain engaged in the current dialogue about quality-standards and evidence within the context of the role of higher education in society. Faculty and their peers need to judge the entire academic enterprise, which includes the teaching and learning environment, the competencies of the faculty, and the governance of the institution and program, all within the context of the mission and goals of the institution and academic program.

Faculty ownership is critical for preserving the essential values and norms of the academy.In a learning community the faculty must assume the responsibility for the improvement of academic quality. A vibrant learning community needs a strong internal assessment and evaluation mechanism for constantly learning about itself. Accreditation must become more proactive in its insistence that faculty themselves are involved in learning what and how well they do their work, using student performance as one major criterion of quality. Accreditation thus becomes both an external judge of quality and a check on the institution's internal commitment and mechanism of monitoring quality. In the final analysis, the challenge is one of trust-can the faculty at the local institution, reinforced by a collegial process of accreditation, meet the accountability demands of the larger public?

In summary, standards and evidence linked to student learning and development have considerable merit and utility both in accreditation and in fostering a learning community. In both, an emphasis on high expectations for student learning and development is a powerful strategy because it points to the centrality of the student in a learning community. But we also must be prudent in our reliance on student performance, keeping in balance the positive and the unintended negative consequences of focusing only on external assessments of student learning and development in defining quality and effectiveness. We will enhance student learning if we collect a wide range of evidence and not only easy-to-collect, uniform, or standardized performance measures and tests to indicate quality. We will improve the educational environment by recording and studying the connections among resources, student experiences, and student performance and learning. And we will advance the learning community by better understanding student learning, which faculty are in the best position to do.

Developing better assessments of student learning and development and knowing when and how to use them are critical, to be sure. The assessment of student learning can focus faculty discussion about student learning within a learning community and pointing to the need to improve the academic community. Better assessment is one part of the solution to improved student performance. We like the advice of Mr. Lemann, who was quoted in a recent New York Times article (May 4, 1997, p. E4), "You can't fix education by fixing testing." Evidence about student learning is needed to enhance a learning community. The major factor, however, still rests on the faculty and staff at each college to be effective in their teaching, mentoring, and concern for the student.