Toward Instruments of Assessing Sustainability Knowledge: Assessment development, process, and results from a pilot survey at the University of Maryland
Abstract
Colleges and universities strive to educate all students for a sustainable future; however, few institutions assess students’ knowledge of sustainability concepts. Hundreds of institutions are currently measuring their overall sustainability performance using the Sustainability, Tracking, Assessment, and Rating System (STARS), which offers a boost to an institution’s overall sustainability rating if that institution conducts a “sustainability literacy assessment.” Largely due to the popularity of STARS, many faculty and staff who are involved with campus sustainability management are seeking an easy-to-replicate assessment process and instrument. Researchers at the University of Maryland developed and conducted a sustainability knowledge assessment to meet the needs of their campus and to contribute a model for the greater higher education community. This paper shares the development process, assessment instrument, significant findings, and recommendations for campuses seeking to conduct their own assessment. Correspondence should be directed to the University of Maryland – Office of Sustainability at sustainability@umd.edu
Keyword:
Introduction
Today, sustainability is shaping both physical infrastructure and curriculum planning on college campuses across the country as faculty and administrators work to provide students with the knowledge and skills they will need in their personal and professional lives. Increasingly, educators and administrators realize that students need both content knowledge and critical thinking skills to address both global and local challenges. At the center of this discussion is how to provide students with the tools they will need to address the linked problems of environmental and economic development. Sustainable development is one way in which these challenges can be met simultaneously as current and future leaders of communities, nations, and international institutions support the growth of strong and robust systems that support the world’s population. Sustainability takes on even more significance with the projection of 9 billion people on the planet by 2050. At the same time, global environmental resources are either finite or, if renewable, many are depleted faster than can be replaced. This unprecedented scale of linked human social problems lies at the center of the push for an increase in sustainability themed courses and activities and other related “green” development initiatives at the collegiate level.
“Sustainability” and the concept of sustainable development came to the forefront of civil international society in 1987, with publication of the United Nations’ book Our Common Future, or the Brundtland Report, as it is commonly referred to. Within this groundbreaking report, sustainable development is defined as, “development that meets the needs of the present without compromising the ability of future generations to meet their own needs” (Brundtland, 1987). While the term sustainability is not defined within the report as a stand-alone concept, in the twenty years since the United Nations Conference on Environment and Development (Rio Summit), scholars and practitioners now accept widely that sustainability is a multi-stakeholder, international process that focuses on the interconnectedness of society, the environment, and economics; further, sustainability pays close attention to the economic and social injustices faced now and in the future by communities around the world.
In the two decades since publication of the Brundtland Report, governments, non-governmental organizations, some businesses, and educational institutions have adopted missions, task forces, and initiatives that integrate sustainability into their work. For higher education, this infusion of sustainability appears in both physical infrastructure and curriculum. Such efforts shape operations at specific colleges and universities but are also reflected across the higher education sector through umbrella initiatives. One such initiative occurred in 1990 when administrators at institutions of higher education composed and signed the Talloires Declaration. This declaration identified and advocated for the critical need to educate students about the serious threats surrounding unsustainable use of the world’s resources and the inequalities global communities face. Since 1990, over 400 institutions from 53 countries have signed this declaration, committing to incorporate sustainability and environmental literacy into the teaching, research, operations, and outreach missions of their campuses (ULSF, 2008).
More recently, the creation of the American College and Universities Presidents’ Climate Commitment (ACUPCC) in 2007 provides a way for college presidents to commit their institutions to these steps: to measure, reduce, and eventually “neutralize” campus greenhouse gas emissions, while, at the same time, developing student capacity that will prepare these graduates to help society do the same (ACUPCC, 2007). This collective commitment identifies the challenges faced by college presidents, while the ACUPCC institution provides support – both technical and organizational – to tackle this difficult challenge. The ACUPCC has “shifted higher education’s attention on sustainability from a series of individual program efforts to a broader strategic imperative” (Cortese, 2012, p.27), noting that while changes in facilities management may allow a university to approach a zero-carbon state as an institution, a focus on operations does not allow for the development of students to help society to do the same. To achieve the second component of the ACUPCC, administrators must be open to sustainability themes being integrated into the curriculum and throughout the college experience.
Many colleges and universities are well on their way in this second component of the Climate Commitment: student formation through education and campus life activities. A review of curricula at a variety of institutions reveals that courses at all levels of collegiate education have been redesigned to include sustainability content or are organized around sustainability and related themes. Globally, many institutions now feature sustainability and sustainable development majors and minors. In addition, on many campuses, student groups advocate for sustainability through existing structures like student government but also through direct action, including development of new student organizations (M’Gonigle & Starke, 2006). These efforts co-occur with a number of related content areas in classes focused on developing sustainable business practices, testing alternative energy technologies, improving social justice and environmental health, as well as analyzing consumption and waste disposal (Bartlett and Chase, 2004)1.
Students are learning about sustainability outside the classroom, too. Financed through grants and sustainability fees, students roll up their sleeves on low impact development installations like rain gardens, green roofs, vegetable gardens, and renewable energy. These development projects, along with many other initiatives, create campuses that are living laboratories where students take lessons learned from the classroom and test this learning in real-world situations.
Learning is also spreading outside university walls as sustainability-themed courses, as well as courses in other disciplines, use service-learning models to explore problems facing local communities (Bowden & Pallant, 2004; Franz, 2004). The service-learning pedagogy allows students to learn sustainability concepts in the classroom and then provides them the opportunity to apply that knowledge in cooperation with others outside of the academy. These dual and interlaced efforts on pedagogy and structured outreach allow students the opportunity to develop the practical skills needed to be change agents in society.
As sustainability becomes a strong focus for institutions, many colleges and universities are developing and implementing a number of monitoring programs that assess the impact of their sustainability efforts, including the sustainability literacy of their students. While the study of sustainability literacy is nascent, much can be learned from the study of environmental literacy. Environmental literacy has been a focus of educational research since the 1970’s, typically measuring personal knowledge and awareness of the environment, along with an individual’s understanding of environmentally associated problems (Roth, 1992). Also included in environmental literacy definitions is participation in environmental protection and environmental problem solving (UNESCO, 1978). The line that differentiates environmental literacy from sustainability literacy can be difficult to see; therefore, crucial to designers of sustainability work on campus is that sustainability literacy involves the components of environmental literacy listed above, as well as components related to social responsibility and civic engagement (Rowe, 2002); further, being a sustainability-literate individual includes understanding the “interrelationship between people, resources, environment and development” (UN, 1983), as well as understanding that the actions of current generations shape future generations. Therefore, in assessing sustainability literacy, an individual’s understanding of how all aspects of society interact must be assessed and not just an inventory of an individual’s knowledge of the environment and environmental problems.
Assessment of student knowledge, attitudes, and behaviors are common in education research, with numerous studies assessing student environmental literacy at all levels of education, K-12 (McBeth & Volk, 2010; Rickinson, 2001; Roth 1992;) and university (Teksoz, Shain, & Takkaya-Oztekin, 2012). However, few studies attempt to assess sustainability literacy at any level. Some campuses are assessing the sustainability knowledge of students before and after one course, much in the manner of pre- and post-test checks on content (Erdogan & Tuncer, 2009; Hiller, Remington, & Armstrong, 2012), or at the completion of minor or major credits (University of Colorado at Colorado Springs, n.d ). However, at the beginning of this study to the best of the researchers’ knowledge, no research had been conducted to assess the sustainability literacy of the student population at a large university 2 .
The need to assess sustainability literacy is articulated widely among sustainability professionals on college campuses, particularly during development of the Sustainability Tracking, Assessment, and Rating System (STARS) tool, released by the Association for the Advancement of Sustainability in Higher Education (AASHE) in 2009. This rating system was developed as a tool to be used by campuses in tracking their progress in sustainability across these categories: education, research, operations, planning, administration, and engagement. One way campuses can gain credits in the curriculum category of the STARS tool is to assess sustainability literacy on campus. The STARS technical manual states that this must be an assessment of sustainability knowledge and not simply a check on values or behaviors (AASHE, 2012). This manual also allows for flexibility as to if a representative sample of the student population must be surveyed or if an instrument can be a pre/post test after one course. This evaluation component of the STARS tool started a nationwide dialogue among sustainability leaders on college campuses. Leaders agree that they need a sophisticated analysis tool but cannot find models to provide insight into how to start the process, format an assessment, or, at the heart of the problem: determine what kind of questions to ask.
This research article offers a self-reflective case study describing the 2011 pilot of a sustainability knowledge3 assessment at the University of Maryland. Results are shared as a narrative of how the assessment tool was developed, deployed, and analyzed. These results are discussed in light of expected and surprising outcomes. Finally, discussed are recommendations for the integration of sustainability into the higher education curriculum along with recommendations for the implementation of sustainability knowledge assessments within institutions of higher education.
Methodology
1. Institutional Context
The University of Maryland’s Office of Sustainability4, along with the University Sustainability Council, coordinates sustainability efforts across campus. These efforts include minimizing the impact of new campus development, implementing a pre- and post-consumer compost program, and administering a $300,000 Sustainability Fund, which provides grants to students, faculty, and staff to implement sustainability projects. The efforts, along with a number of other innovative projects and initiatives, began largely at the facilities level but now include curriculum and faculty engagement.
Three initiatives that focus on integrating sustainability across the university’s curriculum are the Chesapeake Project, an annual two-day workshop focused on assisting faculty members with integrating sustainability into their courses; the Sustainability Advisors Program, which uses peer-educators to introduce first-year students to sustainability in the classroom; and the Sustainability Studies Minor, which started in the spring of 2012 and is open to any undergraduate student. These programs provide UMD students with opportunities to see how sustainability relates to their current majors and future careers, and provides faculty the support needed to implement innovative sustainability-themed material in classrooms. These programs support the University’s stated commitment to prepare graduates with the skills and knowledge needed to bring about change.
2. Assessment Tool Development
While sustainability has long been part of the University of Maryland’s culture, there has never been an attempt to assess the understanding of basic sustainability concepts among the undergraduate and graduate student population. The Office of Sustainability at UMD undertook this task to gather baseline data about student understanding of commonly recognized principles of sustainability. The items on this assessment tool covered topics and themes based on draft sustainability learning outcomes developed in 2008 by the UMD Climate Action Plan Subgroup on Education and Research (Appendix 1). The assessment contained 15 close-ended questions with an open-ended comment section at the end of the assessment, along with demographic questions related to the student’s college affiliation, race/ethnicity, gender, residency, etc. The UMD Campus Assessment Working Group (CAWG) provided guidance and support in all steps of the assessment development process.
Once the draft assessment tool was created, the survey was reviewed by eleven campus faculty members whose courses or research focus on various aspects of sustainability including environment, society, and economics. This faculty input yielded a revised assessment tool that was then tested with two focus groups: 1) sophomores and juniors in an honors program at UMD (seven participants), and 2) freshmen in a living-learning community that focused on sustainability (15 participants). The focus-group process allowed the researchers to test questions, receive input from students, and review scores to ensure question comprehension and assessment tool effectiveness at measuring student knowledge of sustainability. Minor adjustments were made to the assessment tool before implementation. The complete assessment tool is available in Appendix 2.
3. Study Population
The University of Maryland, College Park is a public, land-grant university. In spring of 2011, there were 26,276 undergraduate students and 9,966 graduate/professional students on campus. The majority of undergraduate students attending the university are Maryland residents with almost half of the undergraduate population living on campus in residence halls. The majority ethnic group on campus is white; with Asian being the next largest group. The four largest colleges within the university, accounting for 60% of the student population, are the College of Behavioral and Social Sciences, the College of Computer, Math and Natural Sciences, the College of Art and Humanities, and the James A. Clark School of Engineering. Table 1 details selected campus demographics for spring 2011, further demographic information about the campus can be found from the UMD Institutional Research, Planning, and Assessment group5.
Table 1-University of Maryland spring 2011 demographics
Source: Data from UMD Institutional Research and Planning, Student Enrollment Spring 2011
https://www.irpa.umd.edu/Enroll/ebm-201101.pdf
Categories |
Percentage (%) |
In-State Residency |
|
Undergraduate |
75 |
Graduate |
35 |
Residence Hall Occupants |
45 |
Race/Ethnicity | |
White |
56 |
Asian |
13 |
Black/African American |
10 |
Hispanic |
<1 |
Popular Colleges | |
Behavioral and Social Sciences |
17 |
Computer, Math, and Natural Sciences |
16 |
Arts and Humanities |
14 |
Engineering |
13 |
4. Assessment Distribution
The Registrar’s Office at UMD generated a random sample (9,170 students, about ¼ of the student population) of undergraduate and graduate students registered during the spring 2011 semester. An email from the Sustainability Manager in the Office of Sustainability was sent to the sample on April 21, 2011; the email contained a brief description of the goal of the assessment along with what participants could expect from the “Sustainability Quick Quiz.” The assessment was anonymous and accessible by a link to the assessment tool hosted at the Survey Monkey website. Participants were asked to complete the survey within two weeks; a reminder email was sent out one week after the original email. At the end of the response period, a total of 1,442 assessments were completed (16% participation rate) with 68% of participants being undergraduates and 32% being graduate students.
5. Scoring
Based on the knowledge and input of Office of Sustainability staff members and campus faculty members, individual question answers were deemed “correct” if they were reflective of greater sustainability knowledge and “incorrect” if they were reflective of less sustainability knowledge. Surveys were scored accordingly with respondents receiving one point for each “correct” answer; there was no loss of points for incorrect answers. Some questions had more than one correct answer; therefore, the total number of possible points was 31. Each assessment received a raw score, which was then converted to a percentage (called “sustainability score” from here on). Recall that this assessment is designed to test sustainability knowledge, as determined by respondent’s responses over the full set of questions.
6. Analysis
To investigate trends in the data, average scores were compared across eight demographic groups: college affiliation, class level, housing, gender, race/ethnicity, residency, expressed overall concern for the environment, and number of related courses taken. These categories were chosen based on current assessment practices on campus, literature in the field of environmental literacy (McBeth and Volk, 2010) along with a desire to explore whether current programming efforts and curriculum on campus shape student knowledge of sustainability. The t-test was used to compare the average score for each group with other groups within each demographic category. This statistical check assesses whether the average of the two groups are statistically different from each other. Scores between groups were considered significantly different if the p-value was less than 0.01.
The authors would like to note that 10% of respondents submitted comments at the end of the assessment. These comments were grouped based on the following themes: positive comments (n = 19), thoughts on what UMD could do to improve sustainability on campus (n = 14), comments about specific questions (n = 25), general assessment design comments (n = 22), and any other comments that did not fit into the above categories (n = 66). Comments were helpful in understanding how respondents viewed certain assessment questions and allowed the authors to gain insight that is not possible to gather from multiple choice questions. These comments will be used as the assessment is adapted for future use.
Results
The first step in assessing the sustainability knowledge of the sample was to find the mean raw score and sustainability score for all respondents. The mean raw score for all assessment respondents was 23 points or a mean sustainability score of 74.9%; the mode for all respondents was 83.8%; the median sustainability score was 77.4%; the range is between 16% and 100%; with a standard deviation of 15.66.
One interesting finding, immediately clear upon inspection of the data set, is that students who self-identified as graduate students (master and doctoral levels) scored significantly higher (mean sustainability score = 77%) than all students who self-identified as undergraduate students (mean sustainability score = 74%, p = 0.002). Even more interesting, though, is this finding for undergraduates: no significant difference was found when comparing the mean sustainability score of the four levels of undergraduate education. Table 1 displays the mean sustainability score and participation level as they relate to a respondent’s level of education.
Table 1- Mean scores of respondents by self-identified academic level at the University of Maryland
Education Level |
Mean Sustainability Score (%) |
n |
Freshmen |
74.3 |
193 |
Sophomore |
73.9 |
233 |
Junior |
73.7 |
279 |
Senior |
74.1 |
273 |
Masters |
76.7 |
241 |
Doctoral |
77.1 |
217 |
Since education level does not seem to affect an undergraduate respondent’s sustainability score, the researchers wanted to assess other factors that may affect a student’s sustainability knowledge. One such factor for investigation concerns the school or college within the university in which the respondent is enrolled. The team wanted to assess if significant differences in sustainability scores were found for students in colleges and schools that traditionally have a strong academic focus on sustainability writ large, such as agriculture and natural sciences or life sciences, as compared to colleges that traditionally have less of a focus on sustainability, such as the arts and humanities. Note: students were not asked to identify their specific major but asked to indicate the colleges or schools that they see themselves as a part of.
At the level of colleges and schools, the mean sustainability score was analyzed for all respondents who reported being a part of that particular program. As shown in Table 2, the range in mean scores for the undergraduate students in the 11 colleges was 69% to 80%, with the highest scoring schools being the College of Agriculture and Natural Resources and the School of Architecture and Planning. The lowest scoring colleges were the College of Education, the School of Business, and the School of Public Health. The range in mean scores for graduate students in the 13 colleges was 71% to 83% with the highest scoring schools being the College of Arts and Humanities and the School of Public Health. The lowest scoring colleges were the School of Engineering and the Business School.
It is important to note that since many students have double majors and/or minors in different colleges, determining a student’s main academic focus is difficult; therefore, significance levels comparing mean scores between colleges will not be reported here. Typically, many students see themselves “belonging” to more than one academic unit due to double majors and even minors; therefore, this common condition will remain a technical challenge for all such instruments. None-the-less, these aggregated scores can be used as baseline data for assessing change in student sustainability knowledge at UMD over time. Since the university rolled out a new general education curriculum in fall 2012, this data could be used in the future as colleges and departments implement new or revised courses that focus on the principles of sustainability as outlined in the campus’s proposed Sustainability Learning Outcomes.
Table 2- Mean scores for respondents as self-identified belonging to the 13 academic colleges or schools at the University of Maryland (2011). * These colleges only have graduate level programs.
College |
Mean Sustainability Score (%) |
|||
|
Undergraduates |
n |
Graduates |
n |
A. James Clark School of Engineering |
77 |
132 |
71 |
64 |
Agriculture and Natural Resources |
80 |
56 |
77 |
14 |
College of Arts and Humanities |
74 |
165 |
81 |
42 |
College of Behavioral and Social Sciences |
72 |
188 |
79 |
44 |
College of Computer, Math and Natural Sciences |
75 |
176 |
81 |
100 |
College of Education |
69 |
26 |
74 |
57 |
College of Information Studies (iSchool6)* |
—– |
— |
79 |
29 |
Letters and Sciences7 |
74 |
62 |
73 |
3 |
Philip Merrill College of Journalism |
77 |
12 |
78 |
3 |
Robert H. Smith School of Business |
71 |
100 |
71 |
51 |
School of Architecture, Planning, and Preservation |
80 |
17 |
77 |
14 |
School of Public Health |
71 |
50 |
83 |
13 |
School of Public Policy* |
—– |
— |
76 |
22 |
Mean across all schools |
74.5 |
77 |
Due to difficulty in assessing each respondent’s affiliation with a particular college, an analysis of the number of sustainability-related courses taken per respondent was conducted to better assess a respondent’s exposure to sustainability themes within the curriculum. This approach addresses the large variety of courses that students can take to fulfill their degree requirements and, as noted earlier, addresses the practice of double majors and/or minors. Table 3 displays mean sustainability scores as they relate to the number of courses focused on the sustainability themes in the assessment. Note: students surveyed self-reported and self-identified sustainability-themed courses taken during their time at UMD (see question 25).
Table 3- Mean scores for students based on number of self-reported/self-identified “sustainability-themed” courses taken
Sustainability-Themed Courses Taken |
Mean Sustainability Score (%) |
n |
0 related courses |
74 |
693 |
1-2 related courses |
75 |
588 |
3+ related course |
80 |
157 |
In comparing the scores of respondents who reported taking zero or one-to-two courses, researchers found no significant difference in the scores of the two groups (p = 0.595). However, a significant difference in scores was found between respondents who reported taking zero courses and three-or-more courses (p = 0.0001) and between respondents who reported taking one-to-two courses and three-or-more course (p = 0.0003). In other words, students who took three or more sustainability-themed courses appear to have more sustainability knowledge than students who took zero, one, or two sustainability-themed courses. The significance testing as seen in the given p-values suggests that this relationship between sustainability knowledge and a threshold number of courses is statistically significant, and worth closer examination. This idea of threshold is underscored further by rephrasing this finding: students who took one or two sustainability-themed courses were no more knowledgeable about sustainability than students who took no such courses. Researchers conclude that three sustainability-themed courses may be a minimum exposure of sustainability concepts to increase a student’s sustainability knowledge.
One caveat to this threshold relationship concerns other sources of sustainability knowledge aside from the classroom. The researchers acknowledge that for students on the UMD campus, sustainability is not just something they hear about in the classroom but also through a number of campus-wide initiatives. For example, sustainability programming is integrated in locales other than classrooms like dining and residence halls8. Therefore, the team wanted to assess if a student’s housing affected their sustainability knowledge. Table 4 displays the average mean scores for all respondents based on where they lived during the academic year.
Table 4 a – Mean scores based on student housing
*The commons/courtyard apartments are apartment complexes affiliated with the UMD campus, but students are not required to have a dining plan and there is less residential life programming that occurs.
Student’s Housing |
Mean Sustainability Score (%) |
n |
residence hall |
75 |
376 |
Commons/Courtyard apartments* |
75 |
145 |
Greek housing |
70 |
23 |
off-campus housing |
75 |
670 |
family’s home |
74 |
223 |
When comparing the various housing categories, no significant difference in scores was found based on where respondents live. This suggests that campus programming or living-learning programs are either not responsible for gains in sustainability knowledge or that these programs do not focus on sustainability themes.
One caveat, which does seem to affect the course threshold, is a student’s stated level of concern for the environment. As shown in Table 5, students who self-identified as being somewhat or very concerned about the environment scored significantly higher on the assessment than students who self-identified as being neutral (p = 3.14E-08, p = 1.34E-19 respectively), not very concerned (p = 0.0002, p = 4.53E-06 respectively), or not at all concerned (p = 0.003, p = 0.0003 respectively). Researchers note that a majority of respondents reported that they were somewhat or very concerned about the environment (88%). The researchers analyzed the data to see if there was a correlation between the number of courses taken and the level of concern, but no such correlation was found in this data set.
Table 5- Mean scores as reported as respondents self-identified level of concern for the environment
Level of Concern |
Mean Sustainability Score (%) |
n |
Not at all concerned |
54 |
16 |
Not very concerned |
57 |
29 |
Neutral |
64 |
111 |
Somewhat concerned |
74 |
589 |
Very concerned |
79 |
678 |
Another finding from the assessment was the difference in sustainability scores when examined by race/ethnicity. As shown in Table 6, respondents who self-identified as white scored significantly higher than respondents who self-identified as Asian (p = 2.9E-05) and Hispanic/Latino (p = 0.0002). Respondents who self-identified as Black/African American scored significantly higher than respondents who self-identified as Hispanic/Latino (p = 0.002). No significant difference in sustainability score was found between any other racial groups, including no significant difference between respondents who self-identified as being two or more races and any other group. Though the racial breakdown as a percentage of the assessment respondents is not representative of the University of Maryland student population disaggregated by race/ethnicity, the data does support previous findings related to race/ethnicity and level of science literacy (Klein et al., 1997; Muller, Stage & Kinzie, 2001), which is why the information is included here.
Table 6- Mean scores based on respondent’s self-reported and self-identified ethnicity/race
Race/ Ethnicity |
Mean Sustainability Score (%) |
n |
White |
77.7 |
920 |
Black/African American |
76.7 |
111 |
Asian |
75.3 |
262 |
Hispanic/Latino |
71.0 |
81 |
Two or More |
74.8 |
54 |
These results show selected findings related to the sustainability knowledge of students surveyed at the University of Maryland in the spring of 2011. While these findings are of key importance to University of Maryland faculty members and administrators, as well as the wider circle of higher education sustainability-minded professionals, not all of the results will be discussed in the analysis. For example, the difference in sustainability knowledge between graduate and undergraduate students9 and the differences in sustainability score between racial/ethnic10 groups as reported on the assessment will not be discussed. While these differences may be of interest to both the UMD and higher education sustainability community, the following discussion will focus solely on sustainability knowledge as a function of exposure to sustainability content via coursework. Therefore, the analysis will consider sustainability scores related to college affiliation, number of sustainability related courses taken, and student-stated level of concern. Following the discussion are the research team’s recommendations related to developing and administering a sustainability knowledge assessment at the collegiate level.
Discussion and Analysis of Results
While it was difficult to gain a broad understanding of sustainability scores based on a respondent’s main academic department/program due to the difficulty of parsing out double majors and minors, the research team believes that campus administrators and faculty members should to be aware of national trends related to the programs and departments that include sustainability-focused curricula. The nationally-focused “State of the Campus Environment Report 2008” highlights that generally students within departments and programs focused on the natural and agricultural sciences have more exposure to topics related to the environment and sustainability. This report found that most colleges and universities tend to integrate environment and sustainability content into the natural sciences department as opposed to physical science, engineering, or education departments (McIntosh et al., 2008).
To gain perspective on what this means for the young adult population within colleges and universities, it is important to note that only 8% of colleges and universities nationally have an environmental science course requirement for all students (McIntosh et al., 2008). Also helpful to understanding the relationship between science education and sustainability knowledge is the “popularity” of science tracks chosen by college students. Over the past decade, roughly 11% of the degrees conferred each year are in the natural/agricultural sciences (NSF, 2012), while the largest percentage of degrees conferred each year are business degrees (roughly 20%) (NCES, 2011). Therefore, this “stovepiping” of sustainability curriculum content in one area of higher education, namely the natural sciences, creates a systemic shortfall in the goal of educating all college students on the principles of sustainability. Consequently, the vast majority of college students leave college without gaining the knowledge and skills needed to live personal and professional lives that aid in creating a sustainable society. Therefore, administrators and faculty members need to find ways to integrate sustainability topics across the curriculum.
Another important finding related to coursework concerns number of courses taken as a determinant of sustainability knowledge. In other words, course content experienced in a classroom can be thought of as a threshold of curriculum-based exposure. As our results suggest, students must take at least three courses related to the principles of sustainability to have a high level of sustainability knowledge. These results at the University of Maryland differ somewhat from an oft-cited study assessing the environmental behavior of college students conducted by Wolfe (2002). Wolfe’s work found that one course is sufficient for students to increase the number of their environmental behaviors. The major caveat to direct comparison of Wolfe’s work with the Maryland study concerns the subtle but important difference between environmental behaviors and sustainability knowledge. This fundamental difference may explain the relationship in this study between the threshold number for courses and tested outcome of higher sustainability knowledge. Furthermore, green knowledge as a study objective is not the same thing as green behavior. Wolfe did not assess environmental knowledge and the Maryland study did not assess sustainability behaviors.
As an aside regarding the importance of assessing sustainability behaviors, the University of Maryland has looked at environmental behavior in students. The UMD Campus Assessment Working Group (CAWG) collected data on student sustainability behaviors through the following surveys: “University of Maryland Student Survey” (2011 and 2006), and “Beginning Student Survey” (2008 and 2009) (CAWG UMD, n.d). As this behavior-focused information was collected from different cohorts of students than the study under discussion here, relating the data from those University of Maryland studies on sustainability behavior to our findings on sustainability knowledge is not possible.
Another variable associated with higher sustainability scores is student-stated level of concern for the environment. Students who stated that they were somewhat or very concerned about the environment had higher sustainability scores, though no correlation was seen between higher levels of concern and students taking more courses related to sustainability. In other words, there was not an interactive effect of extremely high sustainability scores for students who self-reported these two attributes: have taken three or more courses with sustainability content and expressed a high level of concern. One caveat that can affect the analysis of the connection between level of concern and sustainability knowledge is that level of concern is subjective; therefore, students taking the survey must first decide what researchers mean by level of concern for the environment and then select what they believe matches their personally-expressed level of concern for the environment. While research suggests that level of concern can be a determinant for taking action to care for the environment (Chawla, 2010; Monroe, 2003; Teksoz, Shain, & Takkaya-Oztekin, 2012), it is difficult to assess how a student’s level of concern increases their level of sustainability knowledge. Another plausible determinant of sustainability knowledge concerns experience (Chawla, 2010). Students may experience a non-classroom event that triggers an increase in their level of concern, such as poor water quality in their hometown, decreased health due to environmental factors, or growing up on a small family farm. Such events may shape course selection in both type and number but the data does not clarify if these events always motivate students to learn more about environmental or sustainability subjects. This is an open question for young adult development: do such life events lead students to integrate lived experience with college course options? Researchers think that looking at this question for sustainability knowledge would be ideal for campus focus group work or other methods of social science inquiry on attitude formation, particularly in young adults.
Recommendations for Creating an Assessment Tool
The following are recommendations developed by the research team to assist other institutions wishing to develop a similar assessment tool. These recommendations were compiled upon reflection of the assessment creation process, analysis of the data, and gaps in the current, albeit quickly emerging, literature.
The first recommendation is that the assessment must be inclusive of all aspects of sustainability. Therefore, when an institution decides to develop such an assessment, key members of the campus sustainability community must be involved in the initial process, i.e. staff members, students, faculty members, and administrators whose work focuses on sustainability. This will ensure that all facets of sustainability are included in the survey. Once the team is convened, it must work together to develop questions that assess all components of sustainability and relate to the environment of the campus community. It is also imperative that the assessment team determine a set of common terms and language to remove specialist terms or jargon from the assessment so that the tool is accessible for all participants.
The second recommendation is that in order to gain the best picture of a respondent’s overall sustainability literacy, the assessment should include questions that gauge their awareness, sensitivity, knowledge, level of concern, and level of responsibility as related to sustainability (Teksoz, Sahin & Tekkaya-Oztekin, 2012). While this approach will lead to a longer assessment, doing so will begin to provide the knowledge needed to better understand the connection between knowledge about sustainability and associated sustainability behaviors. It is important to ensure that the assessment focuses on all aspects of sustainability, not just the environment therefore, questions should focus on aspects of civic involvement, consumption patterns, as well as the environment. Lastly, to fully understand the nuances of sustainability literacy, institutions should incorporate focus groups in the assessment process. While focus groups are resource intensive, they can be the most effective way to gain information that is not easy to parse out from an assessment of this type. Aside from being able to see connections between sustainability knowledge and behaviors, focus groups would also allow researchers to better understand when and how students are learning sustainability knowledge, i.e. in the classroom, around the campus, or through outside sources. Thus, allowing faculty members and administrators to not only assess what students know and how their knowledge changes as the University implements new curricula and new initiatives but also how they can revise courses to match how students prefer to gain new knowledge.
The third recommendation is to adjust the demographic question about college affiliation so that it allows respondents to prioritize their areas of study; this adjustment would aid in better handling double majors/minors. By allowing respondents to select their primary academic college and then have the opportunity to select their secondary academic affiliation, researchers could better understand the connection between academic disciplines and sustainability knowledge. This would then allow an institution to evaluate how the coursework and requirements of various programs influence the sustainability knowledge of students. It is also important that the assessment reflect the general education requirements of the institution. If an institution does have specific requirements related to sustainability-themed coursework, then researchers would be wise to include questions that assess the impact that those courses have on respondents’ sustainability knowledge.
Lastly, researchers and institutions should consider the following technical points. First, while using an online assessment tool that can be emailed to participants makes the assessment easy to administer and evaluate, this venue can be difficult in securing a high participation rate, due to the impersonal nature of the distribution. It is also important to note that the thematic topic of such an assessment can also serve as a hindrance to gaining a complete picture of the sustainability knowledge of a campus, as some people will not open the survey because they do not have an interest in the topic. Therefore, if a campus has an institutionalized assessment tool that is largely successful, researchers interested in assessing sustainability knowledge are encouraged to seek out space on those assessment tools. Or as pointed out by AASHE’s Call to Action, institutions that participate in the National Survey of Student Engagement can add questions related to sustainability to their institution’s survey (AASHE, 2010). Second, while an assessment tool may not be perfect, it does serve as an educational experience, and should be viewed as such during its development. This means participants should be provided with ample information to allow them to fully understand what the question is asking. Also, after a participant submits their responses, they should be provided with resources where they can learn more about the topics discussed within the assessment. A review of the comments from this assessment indicates that many respondents were interested in learning more and many wished to know the “correct” answers.
As noted throughout this report, there are a number of limitations associated with performing an assessment in this manner. While it is important to be aware of these limitations, they should not serve as barriers to developing and implementing a similar assessment at other campuses. Awareness of the limitations of conducting a study in this manner allows for dialogue among institutions and organizations to aid in the development of more sophisticated assessment tools. Despite difficulties with this or any similar instrument, such a process carried out at any institution over time is a self-study discipline that seeks real understanding about relationships and respects the culture of integrity in higher education research. Because sustainability education is rather new, initial steps and program development have relied understandably on reasoned hunches and anecdotes to inform policies. Thus, the wider deployment of such assessment tools can aid administrators, as well as both faculty and staff members, to create a learning environment that fosters not only the discussion of sustainability topics in the classroom but also stimulates the creative thinking and problem solving needed in young people to create sustainable communities.
Acknowledgements: The authors would like to thank the University of Maryland’s Office of Sustainability for their continued support of the assessment. Also to be thanked are the University of Maryland faculty and staff members, and students who assisted during the assessment creation and pilot phases. Finally, the authors acknowledge and deeply appreciate Julie Kromkowski and the University of Maryland Campus Assessment Working Group, for their able and generous assistance in assessment development and distribution.
Notes
1.Up-to-date information related to sustainability initiatives being implemented at institutions around the globe is available through the AASHE Bulletin http://www.aashe.org/connect/enewsletters/bulletin and Blog http://www.aashe.org/blog
2. Since the development of this paper Ohio State University has developed a similar assessment to assess the knowledge of students on their campus. Their assessment is available here http://ess.osu.edu/sites/d6-essl.web/files/imce/Survey%20instrument%20with%2016%20questions%20highlighted.pdf
3. Throughout the assessment development process the researchers acknowledged that the assessment is not a sustainability literacy assessment but is a sustainability knowledge assessment because it seeks to understand participants’ understanding of sustainability principles and does not also seek information regarding individual behaviors, attitudes, or values.
4. To learn more about the programs and projects implemented by the University of Maryland’s Office of Sustainability visit http://www.sustainability.umd.edu/
5. The UMD Institutional Research, Planning and Assessment “Campus Counts” website provides publicly available information about the university including enrollments https://www.irpa.umd.edu/menus.cfm?action=wocngeneral
6. The University reorganized the College of Information Studies during the assessment process; it is now called the iSchool and includes graduate students in Information Technology and Library Science.
7. The College of Letters and Sciences is an “administrative” college that houses students who have not declared a major and not yet of junior status. Once students declare a major, they are moved to an academic college/school
8. Other sources of sustainability programs on the UMD campus include student activism through clubs, pre-professional societies, and student government. These affiliations can be examined in allied assessments efforts including focus groups and assessments of environmental/sustainability group formation on campus.
9. The difference in sustainability scores between graduate students and undergraduate students was not discussed in this paper due to many confounding factors that can lead to the differences in scores found in this study, i.e. previous education, age and developmental stage, life experience, selected major focus, etc.
10. The difference in sustainability scores between racial/ethnic groups was not discussed in this paper due to many confounding factors that can lead to the differences in scores found in this study, i.e. previous education, socioeconomic level, first language, birth country, etc.
References
AASHE. (2010). Sustainability curriculum in higher education: a call to action. Retrieved from http://www.aashe.org/files/A_Call_to_Action_final%282%29.pdf
AASHE. (2012). Sustainability tracking, assessment & rating system (STARS) technical manual 1.2. Retrieved from http://www.aashe.org/files/documents/STARS/stars_1.2_technical_manual.pdf
ACUPCC. (2007). American college and university presidents’ climate commitment- commitment. Retrieved from http://www2.presidentsclimatecommitment.org/html/commitment.pdf
Barlett, P.F, & Chase, G.W. (Eds.). (2004). Sustainability on Campus: stories and strategies for change Cambridge: Massachusetts Institute of Technology.
Bowden, R.D., & Pallant, E. (2004). Maintaining a college-community ecotourism project: faculty initiative, institutional vision, student participations, and community partnerships. In Sustainability on Campus: stories and strategies for change (pp. 197-213). Cambridge: Massachusetts Institute of Technology.
Brundtland, G.H. (1987). Report of the world commission on environment and development: our common future. United Nations. Retrieved from http://www.un-documents.net/our-common-future.pdf
CAWG UMD. (n.d.). Campus assessment working Group- University of Maryland reports. Retrieved from https://www.irpa.umd.edu/CAWG/Reports/
Chawla, L. (2010). Significant life experiences revisited: a review of research on sources of environmental sensitivity. The Journal of Environmental Education. 29(3), 11-21. DOI 10.1080/00958969809599114
University of Colorado at Colorado Springs (n.d) Minor in sustainable development assessment. Retrieved from https://stars.aashe.org/institutions/university-of-colorado-at-colorado-springs-co/report/2011-01-28/1/3/22/
Cortese, A.D. (2012). Promises made and promises lost: a candid assessment of higher education leadership and the sustainability agenda. In The Sustainable University: green goals and new challenges for higher education leaders (pp.17-31). Baltimore: The Johns Hopkins University Press.
Erdogan, M., & Tuncer, G. (2009). Evaluation of a course: education and awareness for sustainability. International Journal of Environmental and Science Education. 4(2), 133- 146. Retrieved from http://www.ijese.com/IJESE_v4n2_Erdogan.pdf
Franz, A. (2004). Restoring natural landscapes: from ideals to action. In Sustainability on Campus: stories and strategies for change (pp. 229-240). Cambridge: Massachusetts Institute of Technology.
Hiller, K.Y., Remington, S.M., & Armstrong, C.M. (2012). Assessing systems thinking skills in two undergraduate sustainability courses: a comparison of teaching strategies. Journal of Sustainability Education. 3. Retrieved from http://www.susted.org/
Klein, S. P., Jovanovic, J., Stecher, B. M., McCaffrey, D., Shavelson, R. J., Haertel, E., Solano- Flores, G., & Comfort, K. (1997). Gender and racial/ethnic differences on performance assessments in science. Educational Evaluation and Policy Analysis, 19(2), 83-97. Retrieved from http://www.colorado.edu/education/faculty/guillermosolanoflores/docs/Gender%20and%20Racial%20Ethnic%20Differences.pdf
McBeth, W., Hungerford, H., Marcinkowski, T., Volk, T, & Meyers, R. (2008). Report: national environmental literacy assessment project: year 1, national baseline study of middle grades students. Retrieved from http://www.epa.gov/education/pdf/MASTERNELA_Year1Report_081208_.pdf
McBeth, W. & Volk, T.L. (2010). The national environmental literacy project: a baseline study of middle grade students in the United States. The Journal of Environmental Education. 41(1), 55-67. DOI: 10.1080/00958960903210031
McCaffrey, M.S., & Buhr, S.M. (2008). Toward a climate-literate society. Environmental Law Reporter- News & Analysis. 38(12).
M’Ginigle, M. & Starke, J. (2006) Planet U: Sustaining the world, reinventing the university. British Columbia, Canada: New Society Publishers.
McIntosh, M., Gaalswyk, K., Keniry, L.J., & Eagan, D.J. (2008). Campus environment 2008: a national report card on sustainability in higher education. National Wildlife Federation. Retrieved from http://www.nwf.org/Global-Warming/Campus-Solutions/Resources/Reports/State-of-the-Campus-Environment-Report/Read-the-State-of-the-Campus-Environment-Report.aspx#Environmenta
Monroe, M.C. (2003). Two avenues for encouraging conservation behaviors. Human Ecology Review. 10(2) 113:125. Retrieved from http://www.humanecologyreview.org/pastissues/her102/102monroe.pdf
Muller, P.A., Stage, F.K., & Kinzie, J. (2001). Science achievement growth trajectories: understanding factors related to gender and racial-ethnic differences in precollege science achievement. American Educational Research Journal. 38(4), 981-1012. DOI: 10.3102/00028312038004981
NCES. (2012). Chapter 3: postsecondary education. Digest of Education Statistics 2011. Retrieved from http://nces.ed.gov/programs/digest/d11/ch_3.asp
NSF. (2012). Chapter 2. Higher education in science and engineering. Science and Engineering Indicators 2012. Retrieved from http://www.nsf.gov/statistics/seind12/c2/c2h.htm#s2
Rickinson, M. (2001). Learners and learning in environmental education: a critical review of the evidence. Environmental Education Research. 7(3), 207-320.
DOI 10.1080/13504620120065230.
Roth, C. E. (1992). Environmental literacy: its roots, evolution, and directions in the 1990s. ERIC Clearinghouse for Science, Mathematics, and Environmental Education. Retrieved from http://www.eric.ed.gov/PDFS/ED348235.pdf
Rowe, D. (2002). Environmental literacy and sustainability as core requirements: success stories and models. Reprinted from Teaching Sustainability at Universities. Walter Leal Filho, editor, Peter Lang, New York. Retrieved from http://cf.ncseonline.org/EFS/DebraRowe.pdf
Teksoz, G., Shain, E., & Takkaya-Oztekin, C. (2012). Modeling environmental literacy of university students. Journal of Science Education and Technology, 21, 157-166.
DOI 10.1007/s10956-011-9294-3
ULSF. (2008). Association of university leaders for a sustainable future: history. Retrieved from http://www.ulsf.org/about_history.html
UN. (1983). Process of preparation of the environmental perspective to the year 2000 and beyond. Retrieved from http://www.un.org/documents/ga/res/38/a38r161.htm
UNESCO. (1978). Intergovernmental conference on environmental education, Tiblilisi. Retrieved from http://unesdoc.unesco.org/images/0003/000327/032763eo.pdf
Wolfe, V.L. (2001). A survey of the environmental education of students in non-environmental majors at four year institutions in the USA. International Journal of Sustainability in Higher Education. 2(4), 301-315. DOI 10.1108/EUM0000000006026
APPENDIX 1
UMD Climate Action Plan Workgroup, Education and Research Subgroup – Dec 2008
Learning Outcomes for Sustainability Education
Understand:
- The meaning of sustainability (the ability to meet the needs of the present without compromising the ability of future generations to meet their needs)
- The fundamental issues of sustainability, including:
- Modern society’s dependence on fossil fuels
- Human population growth
- Habitat destruction/loss of biodiversity
- Economic development verses economic growth verses (growth is inherently unsustainable because it relies upon a never ending supply of resources through the economic system)
- Perceived connection between material consumption and happiness
- Climate change
- Linear systems verses closed loop systems (only closed loop systems are sustainable)
- Differences between non-renewable and renewable materials
- Limits of Earth’s natural resources
- Increasing demand and diminishing stock of fresh water
- Food (origins, health/nutrition, sustainable agriculture)
- The implications of population growth on the environment, economy, and society
- The concept of a carbon footprint and ecological footprint and the factors that affect both
- That sustainability involves complex social, cultural, political, economic and scientific issues
- The definition of carbon neutrality
- The impact of sustainability in maintaining economic, physical, and social health
Do:
- Live sustainably
- Seek work that will contribute to a more sustainable society
- Engage in an informed conversation on issues of climate change and sustainability
- Calculate one’s own footprint
- Make informed decisions on lifestyle changes
Appreciate:
- The inter-relation between humans and the natural world
- That sustainability is a moral and ethical obligation
- The opportunity to grow our economy with green jobs
- The fragile nature of life on earth
- Individuals’ responsibility and government actions are both needed to solve the climate crisis
APPENDIX 2- University of Maryland Sustainability Knowledge Assessment- spring 2011 (in PDF–see top of article).