Sulitest®: A Mixed-Method, Pilot Study of Assessment Impacts on Undergraduate Sustainability-related Learning and Motivation
Mason JSE April 2019 General Issue PDF
Link to the JSE April 2019 General Issue Table of Contents
Abstract: A United Nations international collaboration between the Education for Sustainable Development (ESD) and the Principles for Responsible Management Education (PRME) resulted in the creation of Sulitest® (aka Sustainability Literacy Test) an open, online training and assessment tool freely available to higher education institutions globally. This study analyzes the effectiveness of the newly developed Sulitest® to not only measure sustainability literacy of higher education student populations, but also act as a catalyst for boosting affective learning outcomes by: (a) generating interest in sustainability-related issues, (b) improving sustainability-related understandings, and (c) enhancing students’ interests in the subject matter. In order to do so we present a two-phase, exploratory mixed-method pilot study. Preliminary results from this pilot study reveal Sulitest is a useful tool for not only assessing sustainability literacy but also spurring student interests and motivations in sustainability-related subject matters. Findings, discussion and limitations are provided.
Keywords: sustainability, sustainability literacy, assessment, learning, motivation, Sulitest, higher education
Disclosure: The manuscript author worked among a voluntary cohort of higher education faculty assisting with revisions and updates to Sulitest under the leadership of Debra Rowe, Ph.D., President, U.S. Partnership for Education for Sustainable Development between Oct. 2017 and March 2018.
Increasing population growth, overexploitation of ecosystems, increasing frequencies of natural hazards, and the production and consumption of non-renewable resources are all factors leading to increased interest in global sustainability and sustainable development. Students enrolled in higher education today are tomorrow’s voters, consumers, decision makers, and leaders; therefore, higher education institutions (HEIs) are regarded as essential spaces for promoting social change and sustainable policies (Christie et al., 2013). Higher education plays an important role in exposing students to alternative thoughts on a wide variety of controversial and important subjects that spur debate, and may lead to a more civil world; more fair in its policies of governance and the distribution of resources among inhabitants.
Globally many higher education institutions (HEIs) have created opportunities for diffusing knowledge, values, attitudes and behaviors that favor sustainability. HEIs, as organizations, have also begun integrating sustainability into their mission and vision statements, and further demonstrated their dedication toward advancing sustainability through strategic organizational planning processes, targeted investments in sustainability initiatives, and embedding sustainability frameworks into organizational values as a guide for decision-makers.
Academic scholarship argues for a broad understanding of sustainability that accepts the environment as fundamental while embracing “economic, health, social justice and other humanitarian concerns” (Selby, 2009, p. 103). Sarabhai (2015) holds that “policy instruments and technological solutions are not going to be enough” and that behavioral change is “critical for achieving sustainable development,” (pg. 122). Given such, scholars argue key aspects of sustainable development (SD) should permeate the critical issues global societies face today including: climate change, disaster risk reduction, biodiversity, discussions about market economy and poverty reduction, and sustainable consumption (UNESCO, 2015).
Sustainability literacy is conceived to be the knowledge, skills and mindsets that allow individuals to become deeply committed to building a sustainable future and helps them to make informed and effective decisions toward this end (Sulitest, 2017). Due to the breadth and complexity of the sustainability paradigm the ability for HEIs to measure impacts and progress in this domain has proved challenging. In 2006, in response to a perceived lack of tracking systems for evaluating sustainability progress in higher education, the Higher Education Associations Sustainability Consortium (HEASC) requested the development of a standardized tool for the assessment of higher education sustainability efforts. The resulting product was the Sustainability Tracking, Assessment & Rating System™ (STARS®) available through the Association for the Advancement of Sustainability in Higher Education® (AASHE).
STARS is a ‘transparent, self-reporting framework’ for colleges and universities to measure their organizational performance, and provides higher education institutions points/credit for assessing student sustainability literacy. Currently STARS 2.1 has 135 criteria across four categories. Education & Research, Operations, Planning Administration & Engagement and Innovation. AASHE has produced four versions of the evaluation criteria since 2007. HEIs that submit a self-assessment report using the STARS criteria are recognized by a platinum, gold, silver, bronze or reporting institution standing.
This study analyzes the effectiveness of the newly developed Sulitest® assessment tool, acceptable in the STARS 2.1 AASHE reporting system within the Education & Research category, to not only measure sustainability-related literacy, but also act as a catalyst for boosting affective learning outcomes by: (a) generating interest in sustainability-related issues, (b) improving sustainability-related understandings, and (c) enhancing students’ interests in the subject matter. In order to do so we present a two-phase, exploratory mixed-method pilot study. The following section includes: an overview of sustainability-related pedagogy, a review of sustainability-related assessments, and presents the research questions advanced in the study.
Sustainability Pedagogy & Practices
Prior research indicates that post-secondary education makes a limited contribution to changes in graduates’ knowledge and attitudes toward environmental and sustainability issues (Arnon, Orion, & Carmi, 2015; Yavetz, Goldman, & Pe’er, 2009). Yet, a variety of pedagogical approaches and learning outcomes have been recognized as contributing to broadening sustainability literacy in higher education student populations. Participatory, active, and experiential learning are argued to be essential strategies for promoting deep and meaningful learning in higher education (Dori & Belcher, 2005; Tsaushu, et al. 2012). Specifically, in the field of sustainable development (SD) “experiential, interdisciplinary, collaborative, and student-centered learning” are highly recommended approaches (Cotton & Winter, 2010; Domask, 2007) that enable students to associate their learning with real-life issues (Higher Education Academy, 2013). These efforts often include opportunities for personal involvement. These student contributions toward the learning process often include activities and assignments that involve service learning, civic engagement, group discussions, role play, and are all useful in the facilitation of affective learning (Shepard, 2008).
Extant literature also indicates that knowledge and higher order thinking skills are essential, but not sufficient to develop students’ views about SD, and that learning outcomes should include both cognitive and affective components (i.e., values, attitudes) (e.g. Shephard 2008; Sipos, Battisti, & Grimm 2008). Consequentially, pedagogical approaches that provide experiential, interactive, and collaborative opportunities encourage students to develop and reflect on their own values, enabling the educational process to shift from a knowledge transfer focus (e.g., cognitive components) toward personal transformation (Higher Education Academy, 2013) thus resulting in increased affective learning (Shephard, 2008). In the SD domain affective learning outcomes may include: awareness, attitudes, and commitment. Affective learning may manifest in the awareness of environmental impacts of human activity, attitudes toward social equity, and the motivation to promote or engage in sustainability behaviors (Domask, 2007; Sipos, Battisti, & Grimm, 2008).
Assessing Sustainability Literacy in Higher Education
As campus sustainability initiatives have expanded so too has interest in how best to assess the progress and impact of these efforts. Callewaert (2018) notes sustainability assessment generally fall into two distinct categories, “those focused on the assessment of student learning regarding sustainability, and those focused on the assessment of campus culture—the sustainability values, behaviors, and awareness of students, faculty and staff,” (pg. 453). Historically, AASHE has supported both categories of assessment, yet there remained a demand for a standardized tool designed to acutely measure sustainability literacy in student populations within higher education.
The 2012 United Nations Conference on Sustainable Development urged that HEIs play an increasingly important role in developing their students’ awareness of global sustainability challenges. An international collaboration between the Education for Sustainable Development (ESD) and the Principles for Responsible Management Education (PRME) resulted in the creation of Sulitest® (aka Sustainability Literacy Test) an open online training and assessment tool available free to HEIs worldwide.
Through the Higher Education Sustainability Initiative (HESI), leaders of HEIs and related organizations, acknowledged the responsibility they bear in the international pursuit of sustainable development. The mission of Sulitest.org is to “expand sustainable knowledge, skills, and mindset that motivates individuals to become deeply committed to build a sustainable future and to make informed and effective decisions,” (Sultitest.org, 2017). In general, sustainability literacy assessment tools are designed to “gauge current levels of knowledge and track changes over time, as well as assess the effectiveness of courses and curricula at meeting sustainability knowledge goals,” (Zwickle, Koontz, Slagle & Bruskotter, 2014, pg. 375). These assessment tools are frequently distributed via annual campus-wide surveys, course instructors and/or learning management systems (LMS).
Sulitest enables HEIs to assess whether they are producing sustainability literate graduates, while also engaging stakeholders (e.g., administrators, faculty), which may subsequently accelerate the integration of sustainability into higher education standards and learning outcomes across a multitude of disciplines (Décamps, Barbat, Carteron, Hands, & Parkes, 2017). Yet many HEIs are resistant to changing current assessment methods for a variety of reasons including: (1) the financial resources previously invested in creating and developing current data collection methods, (2) the inability to analyze long term trends in student populations by switching to a new assessment format, and finally, (3) the perceived lack of control for organizing, interpreting and distributing results required for external reporting agencies. Alternatively, many HEIs have found Sulitest to be a preferable alternative to current practices because of (1) the low-cost, no-cost investment required for implementation and utilization, (2) the ability to localize and tailor questions to specific institutions and student populations, (3) the acceptance of the tool in the STARS 2.1 reporting system currently available to HEIs participating in AASHE, and (4) the high-quality of the data analytics and feedback that provides visual insight into HEI’s student populations in a comparative format, both locally and globally.
Sulitest was conceived so that the questions could assess an individual’s current knowledge of sustainable development (SD) and also serve as an information pipeline motivating participants to learn more and act. The stated goal of the overall experience of taking the test is to help learners “understand the bigger picture,” as well as, “be touched and inspired by specific stories or facts,” while avoiding the tendencies to memorize lists of facts, figures, issues and challenges without making systemic connections. Problematic issues arise when HEIs use environmental knowledge as an indicator of environmental literacy, sometimes with the explicitly stated purpose of measuring sustainability literacy. Assessment methods that rely on fact-based, cognitive memorization is an acknowledged weakness of many of the environmental knowledge and environmental literacy instruments historically used to document student learning and literacy in the sustainability domain.
Internationally recognized as one of the seventeen featured initiatives of the UN partnerships for Sustainable Development Goals (SDGs), Sulitest is now a contributor to the review of the 2030 Agenda through the High-Level Political Forum (HLPF) (see Table 1 for a review of SD goals). At the time of this manuscript there are 721 public and private institutions, from 61 countries who have completed 85,139 Sulitests. Each Sulitest question is aligned to one or more goals from the U.N. Global Agenda, creating the largest database of citizens’ awareness and understanding of global sustainability in human history.
Sulitest is designed with (1) a systemic educational framework; (2) uses a list of tags and keywords to ensuring balance among the SD subject matter presented; and (3) provides a direct correspondence between the assessment questions used for assessment and the UN SDGs’ framework. An example question along with feedback is included in the appendix materials of this manuscript. The Sulitest Core Module uses 30 questions from the same question bank for every respondent worldwide. An additional 20 questions are localized to specific countries and cultures. The test is currently is available in a variety of languages including: English, Spanish, Chinese, Japanese, French, Portuguese, Welch, and Italian.
| Table 1. UN Sustainable Development Goals (SDGs)- 2030 Agenda |
| SDG1 End poverty in all its forms everywhere |
| SDG2 End hunger, achieve food security and improved nutrition and promote sustainable agriculture |
| SDG3 Ensure healthy lives and promote well-being for all at all ages |
| SDG4 Ensure inclusive and equitable quality education and promote lifelong learning opportunitiesSDG5 Achieve gender equality and empower all women and girls |
| SDG6 Ensure availability and sustainable management of water and sanitation for all |
| SDG7 Ensure access to affordable, reliable, sustainable and modern energy for all |
| SDG8 Promote sustained, inclusive and sustainable economic growth, full and productive employment, decent work for all |
| SDG9 Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation |
| SDG10 Reduce inequality within and between countries |
| SDG11 Make cities and human settlements inclusive, safe, resilient and sustainable |
| SDG12 Ensure sustainable consumption and production patterns |
| SDG13 Take urgent action to combat climate change and its impacts |
| SDG14 Conserve and sustainably use the oceans, seas and marine resources for sustainable development |
| SDG15 Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss |
| SDG16 Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels |
| SDG17 Strengthen the means of implementation and revitalize the Global Partnership for Sustainable Development |
| Note: The establishment of the United Nations High-level Political Forum on Sustainable Development (HLPF) was mandated in 2012 by the outcome document of the United Nations Conference on Sustainable Development. HLPF is the main United Nations platform on sustainable development and it has a central role in the 2030 Agenda for Sustainable Development and Sustainable Development Goals (SDGs) at the global level. For more information visit: https://sustainabledevelopment.un.org |
Given the stated objectives of Sulitest, and in consideration of the format and elements integrated into the Sulitest design, this analysis seeks to understand if the assessment tool is effective at both assessing sustainability literacy and also delivering content in a manner that positively impacts sustainability-related awareness and attitudes (e.g., affective learning outcomes). In order to do so the below research question is offered.
RQ1. Does utilization of the Sulitest impact (a) student learning and (b) student motivation toward sustainability-related concepts?
In October 2017, the Higher Education Sustainability Initiative’s (HESI) Sustainable Literacy Test (Sulitest) reports that currently students’ knowledge about the Sustainable Development Goals (SDGs) is relatively homogenous across all 17 SDGs, with some variations noted. A recent report titled, ‘Mapping Awareness of the Global Goals,’ found that there are no SDGs on which students had complete awareness and no SDGs that had a very low level of awareness (Decamps & Carterton, 2017).Overall students had the highest awareness on SDG 2 (zero hunger), SDG 8 (decent work and economic growth), SDG 11 (sustainable cities and communities), SDG 14 (life below water), SDG 15 (life on land) and SDG 16 (peace, justice and strong institutions), with scores of 60 percent or greater on questions related to these six SDGs.
Sustainability principles have broad applications across many academic domains of the natural and social sciences, in addition to the arts, business, education and technology. As a result some academic programs may emphasize sustainability concepts to a greater degree compared to their counterparts. Because sustainability may be emphasized in some academic programs more than others a student’s perceived knowledge of sustainability is anticipated to moderate responses to Sulitest and the below research question is offered.
RQ2: Will a priori sustainability-related knowledge moderate learning and motivation outcomes?
Method
In the fall of 2017, undergraduate students from a Midwest institution were invited to take part in a 2-phase exploratory pilot study which involved completing Sulitest, and a follow-up survey of the participants’ attitudes and motivations toward sustainability. To begin basic demographic information was collected including: age, ethnicity, and educational status (i.e., freshman, sophomore, etc).
Perceived Knowledge. Participants were asked to respond to a single item to measure their perceived knowledge of sustainability on a 7-pt Likert scale ranging from far above average to far below average.
Functionality. Several attitude statements were used to measure participants’ attitudes toward Sulitest including: ease of registration, clearly use of language, page by page question style, easily understood information, preference for online format, and finally, length/time required for completion. Each statement was assessed on a 7-pt Likert scale ranging from strongly agree to strongly disagree.
Learning. A series of attitude statements were used to gauge the impact of the assessment tool on learning such as: The Sulitest helped me to: (a) reflect on my sustainability knowledge, (b) understand my growth and improvements, (c) see sustainability opportunities in the world around me, (d) understand how my knowledge of sustainability compares with others in the U.S., and (e) understand how my knowledge of sustainability compares with others globally. Each response was assessed on a 7-pt Likert scale ranging from strongly agree to strongly disagree.
Motivation. A series of attitude statements were used to measure the impact on student motivation by responding to a series of statements such as: The Sulitest motivated me to: (a) seek additional information about sustainability-related concepts, (b) share sustainability-related information with others I know, and (c) will retake the Sulitest voluntarily in the future. Each statement was assessed on a 7-pt Likert scale ranging from strongly agree to strongly disagree.
Qualitative Questions. Three open-ended questions were provided to participants. Each participant was invited, but not required, to respond. The questions asked participants to describe the perceived strengths of Sulitest, describe the value of their experience, and each was invited to offer critique and feedback for improving or augmenting Sulitest.
Report of Findings
A total of 45 undergraduate participants were recruited through a Midwestern university in the United States. Participants completed a two-phase study in the fall of the 2017. The first phase of the study required students to complete the Sulitest. The second phase of the study included a follow-up survey, distributed through the Communication Research Lab (CRL) using Qualtrics software within seven days of Phase 1. The materials and measures used in this study were approved through the Institutional Review Board (IRB) and findings will be presented in two sections. The first section summarizes the Sulitest results for this sample, and the second reports the quantitative and qualitative results of this pilot study, including the Phase 2 14-item scale representing the three areas of concentration measuring attitudes and perceptions toward functionality, learning and motivation.
Sample Description
The sample was 71% female (n=32) and 29% male (n=13). The sample was 80% White/Caucasian (n=37), 13% African American, 4.35% American Indian/Native American and 2.17% identified as more than one race. The sample was 2% freshman (n=1), 11% sophomore (n=5), 31% juniors (n=14), 49% seniors (n=22), and 6% graduate students (n=3).
Phase 1: Sulitest Feedback for Sample
The Sulitest feedback indicates that on the Core International items participants scored well on in the following areas: sustainability, humanity and ecosystems; global and local human construction systems; transition toward sustainability; and finally, role-to-play, individual and systematic change. The strongest responses, represented by the highest degree of accuracy related to the SDG 14 Life below water, SDG08 Decent work and economic growth, SDG17 Partnerships for the Goals, and SDG12 Responsible Consumption and Production.
Phase 2: Quantitative Results
In order to answer the RQs advanced in the pilot study a quantitative analysis was conducted. This section reviews the Phase 2 post-Sulitest data gathered using the 14-item scale designed to assess the functionality, learning and motivation. Table 2 below reports these findings.
Functionality. Sulitest functionality was a strength overall. Students reported strong, positive responses to the online format (M=6.21, SD=1.25), ease of registration (M=5.86, SD=1.25), and clarity of language (M=5.95, SD=.89).
Learning. Results show participants reported the Sulitest helped them to reflect on their own sustainability knowledge (M=5.63, SD=1.40), learned to see sustainability-related opportunities in their environments (M=5.33, SD=1.34) and also helped them learn how their sustainability knowledge compares to their peers in the U.S. (M=5.51, SD=1.10) and globally (M=5.30, SD=1.01).
Motivation. In this pilot study we narrowly focus on motivations to seek and share sustainability-related information. Findings reveal that students who completed the Sulitest report being motivated to seek additional information from others they know (M=4.60, SD=1.43) and also share that knowledge with others they know (M=4.56, SD=1.30).
| Table 2. Sulitest® Impacts on Undergraduate Student Learning & Motivation | |||
| M= | SD= | ||
| Functionality | I prefer page by page questions | 4.00 | 1.38 |
| Sulitest provides easy to understand information for each question. | 5.16 | 1.53 | |
| The online format is preferred to a paper version | 6.21 | 1.25 | |
| Sulitest took too much time to complete | 4.37 | 1.63 | |
| The language used in the Sulitest was easily understood | 5.95 | .89 | |
| The registration process for the Sulitest was easy to understand | 5.86 | 1.25 | |
| Learning | The knowledge I gained from Sulitest helped me to see sustainable opportunities around me. | 5.33 | 1.34 |
| I learned a great deal by completing the Sulitest. | 5.09 | 1.39 | |
| The Sulitest feedback helped me to reflect on my sustainability knowledge | 5.63 | 1.40 | |
| The Sulitest helped me to understand how my knowledge of sustainability compares to others in the U.S. | 5.51 | 1.10 | |
| The Sulitest helped me to understand how my knowledge of sustainability compares to others globally | 5.30 | 1.01 | |
| Motivation | The Sulitest motivated me to share sustainability-related information with others that I know. | 4.56 | 1.30 |
| The Sulitest motivate me to seek additional sustainability information from others that I know. | 4.60 | 1.43 | |
| I will likely retake the quiz voluntarily in the future to see if I have improved my sustainability-related knowledge. | 4.12 | 1.87 | |
| Note: n=45, items measured on a 7-pt. Likert scale ranging from strongly agree to strongly disagree with higher numbers indicating stronger levels of agreement; p<.05 | |||
RQ2 sought to understand if students’ perceived knowledge about sustainability would impact their responses to the learning and motivation scale items. The mean score of the perceived knowledge items was 4.00. This scale item was transformed and computed into a new variable with numbers exceeding 5 representing high perceived knowledge and remaining values below 5 representing low perceived knowledge. Scales items were summed for each dimension. In order to answer RQ2 a univariate analysis of variance was computed with knowledge as the fixed factor and learning and motivation as the dependent variables. Results indicate there were no significant differences between the groups F(4, 38)=.67, p=.61. Results are presented in Table 3 below. The lack of a significant statistical difference was disappointing, and likely attributable to the small sample size involved in the pilot study.
| Table 3. Differences based on Current Perceived Sustainability Knowledge | ||||
| High Knowledge | Low Knowledge | |||
| M= | SD= | M= | SD= | |
| Motivation | 4.76 | 1.54 | 4.27 | 1.32 |
| Learning | 5.24 | 1.21 | 5.39 | 1.10 |
| Functionality | 5.39 | .69 | 5.51 | .67 |
| Note: n=45, items measured on a 7 pt. Likert scale ranging from strongly agree to strongly disagree with higher numbers indicating stronger levels of agreement. | ||||
Qualitative Responses
In addition to the quantitative analysis reported above, participants were also invited to respond to three open-ended questions. Participants were asked to describe in their own words the favorable attributes of Sulitest, the perceived impact on student learning, and were encouraged to offer critiques or recommendations for improvement. Responses were reviewed and categorized according to emergent themes. The findings from these responses are described in more detail below.
Favorable Attributes.
Two themes emerged in the qualitative responses about Sulitest format and design including: the ease of completing the assessment and the nature of the feedback. Students frequently reported ease toward multiple aspects of this assessment tool. For example, some noted the ease in understanding the test, ease of navigation, and the ease of following the instructions and directions incorporated into Sulitest.
The second theme in this category involved the quality of feedback provided through Sulitest to the student. The immediacy of the feedback (i.e., viewing correct answers) was repeatedly noted, but more specifically the descriptive explanations for the correct and incorrect choices as well. Although the immediacy and timeliness of the feedback was a noted strength, receiving the explanations while proceeding through the test was commonly cited as having an impact on learning. The final attribute noted by participants was the comparative feedback they were given upon Sulitest completion. Sultitest contrasts participants’ scores to others in the U.S. and across the globe, and makes this information available to Sulitest participants upon completion of the survey.
Learning Impacts
Two themes emerged in this category including: benchmarking and self-reflection. We loosely use the term benchmarking in a non-organizational, managerial sense that narrowly refers to the process students engage in to compare their knowledge and skills with unknown members among their learning communities. Student learning through benchmarking was a reoccurring theme. The benefit of being able to see their individual scores in a comparative context allowed them to understand their own strengths and weaknesses, in addition to understanding how the scores of ‘others’ like them, both locally and globally. Many reported their results as being “eye-opening,” this benchmarking process was the most widely cited impact on learning.
The self-reflection theme, or thoughts about one’s character, actions, and motives, is an umbrella term used to encapsulate a variety of responses. For example: “I felt confident with some answers and it allowed me to reflect;” “It made me realize how much more I have to learn in regards to sustainability both in the United States and around the world,” and “I will look deeper for more sustainable ways to go about my everyday life.”
Critique & Improvements
All participants were invited to offer critiques and recommendations for improvement with Sulitest. There were very few responses to this question; however, there were two areas that repeatedly appeared in the content: length and interactivity. The first suggestion was to improve interactivity by possibly using different response options that would allow subjects to respond other than selecting from multiple choice questions. Interactivity may be improved by allowing respondants to move a toggle along a percentage plain on their device screens, opposed to selecting a percentage for a multiple choice list. Two participants noted a “choose your own adventure” format that would allow participants to select from multiple options within a SDG content category, so there is active decision-making guiding participants’ experience.
The second recommendation relates to the length of Sulitest overall. HEIs that use the Sulitest platform have an option to use the Core International items (n=30), along with an additional 20 questions that reflect local, regional, and cultural considerations resulting in a 50-item test length. Paying-institutions are able to create additional assessment items localized to their campus and communities. The length of the overall test should be considered if the inclusion of additional items will significantly extend the time required for completion. If the Sulitest length is extended too long participants may fall into response sets and/or not provide the mindful responses needed to accurately assess sustainability literacy in HEI student populations.
Discussion
A major role of education is to empower citizens so that they are able to face the complex challenges of the 21st century including enabling change, making informed decisions and collectively building a sustainable future. Yet, assessment of progress can be a time-consuming process, that increases faculty or administrative load, and may be costly, especially for underfunded HEIs with little outside support, other than public funding. Sulitest is an assessment resource freely available for HEIs, and private organizations outside of academia, to raise awareness about global sustainability and assess the sustainability literacy of a variety of internal and external stakeholders including: students, staff, faculty, and alumni.
Findings from this study are consistent with previous reports in that students in this sample scored high on specific development goals including: SDG 14 Life below water and SDG08 Decent work and economic growth. This exploratory, pilot study did not find significant differences between those who self-identified as having a high level of perceived sustainability-related knowledge, compared to those with average or lower amounts of perceived knowledge on the factors of learning and motivation. Still, future examinations with larger sample sizes may reveal individual characteristics that moderate the anticipated affective learning outcomes in a more systematic way.
Preliminary results from this pilot study show Sulitest to be a useful activation strategy to both assess sustainability literacy but also spur interests in sustainability-related subject matters. Findings indicate that participation in Sulitest was received positively by students who noted the immediate feedback, correct answers and explanations, aided their learning and understanding of sustainability-related subject matters.
The measured motivations used in this study are limited to seeking and sharing sustainability-related information with others known to the participant. Alternative methods for assessing motivations have been used in previous scholarship. For example, Mintz & Tal (2017) found two distinct motivations: motivation to learn (i.e.: ‘I wish to continue learning on these issues’), and motivation to enhance sustainability (i.e.: ‘following this course I now wish to be more involve in helping these people in finding sustainable solutions for their problems’) (pg. 10). Future research could more fully and systematically approach the subject of motivation in novel and unique ways not currently represented in this study.
This exploratory pilot study has several noted limitations. The first limitation is the lack of measured behavioral outcomes as indicators for possible future sustainability engagement such as: willingness to join or support campus student organizations, attend campus workshops and activities involving sustainability outreach, obtaining a related academic certificate or additional professional credentialing, or interests in future enrollment in sustainability-focused coursework. All possible outcomes motivated by participation in Sulitest. A second limitation results from the methodological design of the pilot study. The Phase 2 data failed to connect the motivation outcomes to the pedagogical approaches that have contributed to students’ who performed highly on Sulitest. Capturing this type of data would be beneficial in order to more fully understand which pedagogical strategies are yielding strong awareness and literacy in higher education student populations. Finally, the sample size in this pilot study was small, and over 70% were females. Moving forward additional research should investigate diverse demographic factors and individual differences in order to more fully understand if there are differential impacts based on psychological and sociological factors. The study of individual differences in sustainability literacy may be particularly useful for private organizations with access to a broader age range of private and public employee populations.
Conclusion
Ull, Minguet & Agut (2014) advance that education in sustainability is vital for future employability in many fields of study and application. Scholars also maintain that an anticipated lack of adequate knowledge will be one of the main obstacles to achieving sustainable development globally. HEIs have recognized the important role they play in educating student populations in the environmental, social and economic aspects of sustainability for the benefit of society as a whole. Preliminary results from this pilot study reveal Sulitest is a useful activation tool for not only assessing sustainability literacy but also fomenting attitudes and interests in sustainability-related subject matters.
Work Cited
AASHE. (2018). Association for the Advancement of Sustainability in Higher Education. Retrieved on March 15, 2018 http://asshe.org
Arnon, S., Orion, N., & Carmi, N. (2015). Environmental literacy components and their promotion by institutions of higher education: An Israeli case study. Environmental Education Research, 21(7), 1029-1055. 10.1080/13504622.2014.966656
Callewaert, J. (2018). Sustainability literacy and cultural assessments. In W. Leal Filho, R. Marans, & J. Callewaert, J. (Eds.), Handbook of Sustainability and Social Science Research, pp. 453-464. New York: Springer International Publishing.
Christie, B. A., Miller, K. K., Cooke, R., & White, J. G. (2015). Environmental sustainability in higher education: What do academics think? Environmental Education Research, 21(5), 655-686. 10.1080/13504622.2013.879697
Cotton, D., & Winter, J. (2010). It’s not just bits of paper and light bulbs. A review of Sustainability pedagogies and their potential for use in higher education. In P. Jones, D. Selby, & S. Sterling (Eds.), Sustainability Education Perspectives and Proactive across Higher Education, pp. 39–54. Routledge Publishing.
Décamps, A. & Carteron, J.C. (2017). Mapping awareness of the global goals. Technical Report, July 2017. Retrieved from: Student sustainability knowledge highest on SDGs 2, 8, 11, 14, 15, 16 Premier Official News.
Décamps, A., Barbat, G., Carteron, J.C., Hands, V., Parkes, C. (2017b). Sulitest: A collaborative initiative to support and assess sustainability literacy in higher education. The International Journal of Management Education, 15(2), 138-152
Domask, J. J. (2007). Achieving goals in higher education: An experiential approach to sustainability studies. International Journal of Sustainability in Higher Education, 1, 53–68.
Dori, Y. J., & J. Belcher. (2005). How Does Technology-Enabled Active Learning Affect Undergraduate Students’ Understanding of Electromagnetism Concepts? Journal of the Learning Sciences, 14(2), 243–279.
Higher Education Academy Education for Sustainable Development. (n.d.). Retrieved November 12, 2018 from https://www.heacademy.ac.uk/workstreams-research/themes/education-sustainable-development
Mintz, K., & Tal, T. (2016). The place of content and pedagogy in shaping sustainability learning outcomes in higher education. Environmental Education Research, 1-23. 10.1080/13504622.2016.1204986
Otte, P. P. (2016). Integrating sustainable development in higher education through experience-based learning: Insights from experts in team (EiT) for developing a combined theoretical framework. Journal of Education for Sustainable Development, 10(1), 131-159.
Sarabhai, K. V. (2015). ESD for sustainable development goals (SDGs). Journal of Education for Sustainable Development, 9(2), 121-123. doi:10.1177/0973408215600601
Selby, D. (2009). Towards the Sustainability University. Journal of Education for Sustainable Development, 3(1), 103-106.
Shephard, K. (2008). Higher education for sustainability: Seeking affective learning outcomes. International Journal of Sustainability in Higher Education, 9(1), 87–98.
Sipos, Y., B. Battisti, & Grimm, K. (2008). Achieving transformative sustainability learning: Engaging head, hands and heart. International Journal of Sustainability in Higher Education, 9(1), 68–86.
Sulitest (2017). Sustainability Literacy Test. Retrieved February 20, 2018 from http://www.sulitest.org
Tsaushu, M., Tal, T., Sagy, O., Kali, Y., Gepstein, S. & Zilberstein, D. (2012). Peer learning and support of technology in an undergraduate biology course to enhance deep learning. Cell Biology Education, 11(4), 402–412.
Ull, M. A., Minguet, P. A., & Agut, M. P. M. (2014). Training in sustainability as essential for future employability. Procedia – Social and Behavioral Sciences, 139, 543-550. 10.1016/j.sbspro.2014.08.066
UNESCO. (2015). Education for sustainable development. Retrieved November 12, 2018 from http://www.unesco.org/new/en/education/themes/leading-the- international-agenda/education-for-sustainable-development/
Yavetz, B., Goldman, D., & Pe’er, S. (2009). Environmental literacy of pre-service teachers in Israel: A comparison between students at the onset and end of their studies. Environmental Education Research, 15(4), 393–415.
Zwickle, A., Koontz, T.M., Slagle, K.M., & Bruskotter, J.T. (2014). Assessing sustainability knowledge of a student population: Developing a tool to measure knowledge in the environmental, economic and social domains, International Journal of Sustainability in Higher Education, 15(4), 375-389.
Appendix
Example: Sulitest Core Module Question
“Earth Overshoot Day” is the calculated calendar date on which the current world population has consumed more resources than Earth’s capacity to regenerate those resources. From that day on, humanity is thus using up resources of future generations.
How has the “Earth Overshoot Day” date moved in the last 50 years?
a) Fortunately current world’s population does not overshoot the use of available resources
b) In 2017 it was later in the year than 50 years ago
c) In 2017 it was earlier in the year than 50 years ago
d) It hasn’t moved
e) I’m not sure
If correct, participants receive the below message along with a helpful informational link such as a URL to Earth Overshoot Day for more information. http://www.overshootday.org/newsroom/past-earth-overshoot-days/
You have chosen the expected answer.
Comment: In 2016 the world’s capacity to nourish and sustain the world’s population was exceeded on August 3rd (also called « earth overshoot day »), compared to 1971 when the world’s population and its consumption patterns surpassed the world’s carrying capacity on December 21st . Earth overshoot day steadily moves earlier in the year as population grows and unsustainable consumption patterns spread.
A.M. Mason, Ph.D. currently serves as an Associate Professor in the Pittsburg State University Department of Communication and also teaches in the College of Arts & Sciences Interdisciplinary Studies Sustainability, Society and Resource Management undergraduate degree program. Mason’s academic interests in risk and crisis communication span the public health and environmental contexts. Mason is the director of the Communication Research Lab, co-advisor for the PR/Advertising club, and teaches within the Strategic Communication emphasis within the Department of Communication. 
