Evaluation of the Climate Time Line Information Tool


Cooperative Learning, Evaluation, and Assessment Research (CLEAR) Team

Elizabeth Grassi and Heather Smith
July, 2002


CIRES logoUniversity of Colorado LogoNOAA logo

Table of Contents

Introduction

The Climate Time Line Evaluation
Demographics of Participants
Research Questions and Data Selection
Data Analysis

Findings
Question #1
- Is the website technologically accessible to users?

Survey Findings
Observation Findings
Interview Findings
Expert Opinion

Question #2 -Did the CTL Website increase the 8th, 9th, and 12th grade studentsí understanding of climate science, the relationship between climate and human history, the forces that drive climate variability, and the connection between climate and current events.

Organization of Question #2
Determining appropriate audience

Limited time to view site
Benchmarks of understanding

Section One- 8th, 9th and 12th Grade Students
Climate Science
Climate and Human History
Climate Variability
Climate and Current Events
Was any aspect of the website unclear?
Quiz Findings

Section Two- Undergraduate Students
Climate Science
Climate and Human History
Climate Variability

Quiz Findings

Section Three- Expert opinions

Conclusion

Key Findings
Implications

Appendix

Part One- Interview/Open-ended survey responses
Part Two- Powers of 10 Model
Part Three- Expert Opinion

Tables
Table #1. Survey Question #1a. Is the website easy to use?
Table #2. Survey Question #2a. Organization of website
Table #3. Survey Question #2c. Presentation of materials
Table #4. Survey Question #2a. Helpfulness of links
Table #5. Survey Question #1c. Graphics
Table #6. Survey Question #1d. Trouble Loading Pages
Table #7. Survey Question #1k. I would use this website again
Table #8. Was the website clear and organized?
Table #9: Was the website easy to use and navigate?

Table #10. The website increased my understanding of climate science
Table #11: What scientific information did you learn from the website?
Table #12. What did you learn about climate science?
Table #13. Understanding of the relationship between climate and human history
Table #14. What did you learn about climate history? Survey response
Table #15. What did you learn about climate history? Interview response
Table #16. The website increased my understanding of the forces that drive climate variability
Table #17. What kinds of forces drive climate variability?
Table #18. What kinds of forces drive climate variability?
Table #19. Understanding of the relationship between climate and current events
Table #20. What did you learn about the relationship between climate and current events?
Table #21. What did you learn about the relationship between climate and current events?
Table #22. Was any aspect of the Website Unclear?
Table #23. 8th grade quiz scores
Table #24. Undergraduate quiz scores

Table #25. The website increased my understanding of climate and human history
Table #26. The website increased my understanding of the forces that drive climate
Table #27. The website increased my understanding of climate and current events
Table #28. Undergraduate quiz scores
Table # 29 I understood the Powers of 10 Model
Table #30 . Did you understand the powers of 10 model?
Table #31 Percent of students who accurately explained the powers of 10 model


 

Introduction

Between the Summer, 2001 and Spring, 2002, Mark McCaffrey and Dan Kowal from the Cooperative Institute for Research in Environmental Sciences (CIRES) created the Climate Time Line website, an educational tool designed to provide a climatic context for current weather patterns and a pre-instrumental context for current climatic trends. The website integrates weather, climate, and paleoclimate observations through web-based interface that can: 1) be used for comparing data collected over 10 year, 100 year, and 1000+ year periods, and 2) make such comparisons accessible and meaningful to non-technical users. This website was created for two target audiences: 8-12 students and undergraduate students and their professors.

The Climate Time Line Evaluation

The Cooperative Learning, Evaluation, and Assessment Research (CLEAR) Team at the University of Colorado, Boulder was contracted by Mark McCaffrey at the Cooperative Institute for Research in Environmental Sciences to conduct an evaluation of their Climate Timeline Website. The goal in evaluating the Climate Timeline was to answer two fundamental questions:

  1. Is the site technologically accessible to users?
  2. Does the website increase studentsí understanding of climate science, climate and human history, the forces that drive climate variability, and the relationship between climate and current events?

In the following report, we address these research questions. First, we begin by discussing the demographics of the program participants. Next we examine the methods of data selection and collection. Finally we discuss each research question in detail, including data analysis and findings. Through online surveys observations, interviews and informal feedback, data was gathered to assess the status of the CTL proto-type and help determine the future development of the project. Background information on the evaluation instruments and process can be found on the CTL website at http://www.ngdc.noaa.gov/paleo/ctl/evaluation.html .

In brief, we have found that the Climate Timeline Website is technologically accessible to a wide range of users, and is, in its present state, educationally appropriate for an adult (undergraduate students and professors) audience. However, by adapting the vocabulary and content to suit a younger audience, the website will also be appropriate for 8-12th graders.

Demographics of the participants

The CTL Website was piloted by a total of 70 middle school and high school students, a group of 31 undergraduate students and their professors, and 23 professionals in the science professions.

Middle School and High School Participants

70 middle school and high school students participated in the CTL evaluation. The breakdown of participants is as follows: fourteen 12th grade students, thirty-three 9th grade students, and twenty-three 8th grade students. The middle schools and high school participating in the evaluation serve a diverse group of youth in terms of ethnicity and gender.

Undergraduate/Professor Participants

16 undergraduate science students and 15 professors participated in the CTL evaluation. All students were enrolled in an environmental studies course and evaluated the CTL website as part of their science curriculum. Professors were from various universities around the nation. The gender breakdown of this group consists of 43% female, 23% male and 33% of participants did not indicate gender. Information on ethnicity for this group is not available. This group will be referred to as "undergraduates" in tables for the remainder of the evaluation.

Professionals in the Science Community.

23 adults in the science professions reviewed the CTL website and emailed comments of their reactions to the website to the evaluators. Information on gender and ethnicity for this group is not available.

Research Questions and Data Selection

The CLEAR evaluation obtained data to address the two research questions through both qualitative and quantitative methods, including survey implementation, direct observations of middle school and high school students visiting the website, interviews with participants, and review of open-ended email comments concerning the website. Below we clarify the terms of each question and identify the various data sources.

Question #1: Is the website technologically accessible to users?

The "technological accessibility" of the website was determined by the reported "ease of use" of the website, including the organization of the website, the presentation of the website, the ability to load pages, the ability to find information on the website, and the navigation of the website. The data for this question was obtained through surveys implemented to all participants (8th, 9th, 12th grader, undergraduates and professors), direct observation of participants while visiting the website, interviews with middle and high school participants, and email comments from experts in the field.

Question #2: Does the website increase studentsí understanding of climate science, climate and human history, the forces that drive climate variability, and the relationship between climate and current events?

Whether the website "increased understanding" was determined by the self-reported ability of the student participants (8th, 9th, 12th grader and undergraduate students) to understand the website content, including the science content, the history content, the forces that drive climate variability, and the relationship between climate and current events. The data for this question was obtained through a pre and post quiz, survey implementation to student participants, interviews with middle, high school, and undergraduate participants, and email comments from experts in the field.

Data Analysis

All surveys and quizzes collected for this evaluation were analyzed using an SPSS computer analysis system. This evaluation conducted descriptive statistics to determine the percentage of participants who found the website technologically accessible and to determine the percentage of participants who understood the website content. CLEAR then conducted cross tabulations to determine the relationships between age and time spent on the computer and the level of understanding of the website content. Qualitative data fieldnotes and interview transcripts collected during site visits were analyzed using Spradleyís domain analysis (Spradley, 1980), which identified patterns in the researcherís observations and the interviews with participants. This data helped corroborate, explain, or refute the findings from the quantitative analyses.

Findings

Question #1: Is the CTL Website Technologically Accessible to users?

A: Survey Findings:

A survey questioning the technological accessibility of the website was distributed to each evaluation participant. We examined outcomes of these surveys through two methods: 1) the percent of participants who found the website technologically accessible, and 2) the relationship between age/time spent on the computer and the reported ease of use of the website.

Twenty-three 8th grade students, thirty-three 9th grade students, fourteen 12th grade students, and thirty-one undergraduate students/professors completed the survey. The 8th grade students were given 30 minutes to visit the website before completing the survey, the 9th graders were given 30 minutes to visit the website before completing the survey, the 12th graders were given 30 minutes to visit the website before completing the survey, and the undergraduates/professors visited the website for various times ranging from 11 minutes to 2 hours. Regardless of time spent visiting the website and age of participant, the data show that the majority of participants (i.e., 50% or greater) find the website technologically accessible and easy to use. Tables 1-7 below break down each survey question concerning technological accessibility and show how the majority of participants responded to each question.

Table #1: Survey question #1a. Is the website easy to use? Percentage of participants who marked "agree."

Is the website easy to use?

Survey data indicates that 42.9% or greater of participants "agree" that the website is easy to use.

Table #2: Survey question #2b. The Site was 1 = very well organized, 2 Ė well organized, 3 = not very organized, 4 = not well organized at all. Percent of participants who marked 2 Ė "well organized."

The site was "well organized"

Survey data indicates that the majority (52% or greater) of 9th grade, 12th grade, and undergraduate students report the website to be "well organized. A slightly lower percent of 8th graders (47.8%) also report the website to be "well organized.

Table #3: Survey question # 2c. The presentation of materials was 1 = excellent, 2 = good, 3 = fair, 4 = unsatisfactory. Percent of participants who marked 2 Ė "good"

The presentation was good

Survey data indicates that the great majority (64.3% or greater) of all participants report that the presentation of materials on the website was "good."

Table #4: Survey question # 2a. The links were 1 = very helpful in finding information, 2 = helpful in finding information, 3 = not very helpful, 4 = not helpful at all. Percent of participants who marked 2 Ė "helpful"

The links were helpful

Survey data indicates that the great majority (62% or greater) of all participants report the website links to be "helpful" in finding information.

Table #5: Survey question # 1c. The Website had too many graphics. Percentage of participants who marked "disagree."

The website had too many graphics

Survey findings indicate that the great majority (55% or greater) of all participants report that they "disagree" that the website had too many graphics.

Table #6. Survey question # 1d. I had Trouble Loading Pages. Percentage of those who marked "disagree"

I had trouble loading pages

It is important to note that in table #6, "I had trouble loading pages," both the 9th graders and the 12th graders report a lesser percentage of "disagree" than do the 8th graders and undergraduates/professors. We attribute the 9th and 12th grade responses to the technological problems these participants experienced with their school computers which are likely related to bandwidth limitations of their networks. Both the 9th graders and the 12th graders experienced a shut-down of all computers various times while trying to manipulate the website and, therefore, experienced trouble in loading the web pages. The 8th graders who attended a different school and the undergraduates/professors who attended university had no technological problems with their computers and therefore were able to load web pages rapidly and easily.

Table #7. Survey question # 1K. I would use this website again.

The majority of 8th grade, 12th grade, and undergraduate students/professors either "strongly agree" or "agree" that they would use this website again (60.9% of 8th graders "strongly agree" and "agree," 53.9% of 12th graders "strongly agree" and "agree," and 83% of undergraduates/professors "strongly agree" and "agree."). Note that in Table #7 we see a difference in the responses of the 9th graders. The majority of 9th graders (51.7%) reported that they would not use this website again. We again attribute their response in some part to a high level of frustration concerning the technological difficulties experienced with the school computers.

Summary of Survey Findings

In sum, survey findings indicate that the website is technologically accessible to participants, regardless of age and time spent on the computer. The majority of participants reported that the website was easy to use, the site was well-organized, the presentation of materials was "good," and the links were helpful in finding information. The 8th graders and undergraduates/professors also reported no trouble in loading web pages and reported that they would use the website again. The 9th graders and 12th graders reported more problems loading pages, and a lesser percentage of these participants reported that they would use the website again. We attribute these responses in part to the technological problems these students experienced with their school computers, problems that the 8th graders and undergraduates did not experience.

B: Observation Findings:

Twenty-three 8th grade students, sixteen 9th grade students, four twelfth grade students, and two undergraduate students were observed while manipulating the website. The researchers observed the number of clicks required for each participant to reach their desired web page and recorded participantsí verbal or non-verbal expressions while manipulating the website.

All participants reached their desired web pages in 1-2 clicks. Undergraduate students were observed visiting the website alone and made no verbal expressions. Middle and high school students were placed in pairs for observations and made various comments to each other while visiting the website, such as the following:

"When I was born, I wonder what the stream flow was?"

"What the heck is that?" (referring to a picture of a nebula)

"Right before dawn temperatures are the coldest. I never knew that."

The only participants who exhibited frustration were the 9th and 12th graders when their school computer systems shut-down. For example, in the researcherís field notes, there are excerpts referring to computers shutting down, failing to load pages, and the students experiencing frustration with this situation. However, frustration was expressed about the computer itself, not the Climate Time Line.

Summary of Observations

In sum, we found the observations to corroborate the survey findings. Observations indicate that the website is technologically accessible to all the participants, regardless of age. Participants were able to find the web pages quickly and easily (with 1-2 clicks) and verbal and non-verbal expressions suggested interest in the website content.

C: Interview findings:

Twenty-three 8th grade students, thirty-one 9th grade students, ten 12th grade students, and two undergraduate students were asked to answer either open-ended survey questions or were interviewed concerning the technical "usability" of the website. The breakdown of open-ended survey or interview questions and responses are presented below.

Open-ended survey/interview question #1: Was the website clear and organized?

We asked the 8th, 9th, 12th grade students and undergraduates if they found the website clear and organized. Their responses are indicated below.

Table #8: Was the website clear and organized?

The great majority (70% or higher) of 8th grade, 9th grade, 12th grade and undergraduate students reported the website to be clear and organized. In addition, during interviews many participants provided us with quotes describing why or why not they found the site clear and organized:

8th graders who reported the website to be "Clear and Organized"

"I liked the timeline organized by date. I like the side 1 year, 10 year, and then you can click on the options."

"I could find things easily. I could type something in the search section and I could find something right away."

8th graders who found the website "Organized but not Clear"

"It was organized but not really clear. We couldnít find information very easily. There were not a lot words in the glossary that were in the quiz.""Sometimes I plugged in words into the search and the place that it took me didnít give the information as clearly as I wanted. Maybe it used the word once but didnít have what I needed."

9th graders who found the website "Clear and Organized"

"Pictures were clear and sites explained a lot."

"The glossary was good to find what words meant."

"All the information was put in order and easy to understand"

9th graders who found the website "Organized but not Clear"

"It was easy to find info but none of it made sense."

"It did not have the words I didnít know in the glossary."

12th graders who found the website "Clear and Organized"

"It had a lot of information and broke it down Ė more specific"

"Options were available clearly (such as the search box and site map link)"

Undergraduate comments

"Really easy to search."

"The layout was clear. I liked the links on the side for each page. Each time period had several links which was helpful."

Open-ended survey/interview question #2: Was the website easy to use and navigate?

We asked the 8th, 9th, 12th graders and undergraduates if they found the website easy to use and navigate. Their answers are indicated below.

Table #9: Was the website easy to use and navigate?

The great majority (74% or higher) of 8th grade, 9th grade, 12th grade and undergraduate students found the website easy to use and navigate. In addition, during interviews some of the participants gave us additional comments as to why or why not the website was easy to use and navigate:

8th graders who found the website "Easy to use and navigate"

"It was real easy to use. The glossary and search got us where we wanted to go."

"I thought it was good to navigate because it had easy access to all the sites on the left."

8th graders who found the website "Not easy to use and navigate"

"History and dates were not easy to find. If youíre looking for specific details, there is too much to sift through. Itís easier to find general things."

"The things on the side of the home page helped navigate but were not easy to use."

"There were too many links and each link had a lot of other links and it got too confusing"

9th graders who found the website "Easy to use and navigate"

"The site was easy to use. I could find all the information I needed"

"We could get anywhere we wanted"

9th graders who found the website "Not easy to use and navigate"

"I did not find all the info I needed"

"It was O.K. when it came to use and navigation. It would be better if there were more captions."

9th graders who expressed technical problems with their school computers

"There were problems getting into the program."

"It was fairly easy to use other than the slow downloading"

12th grade comments

"Everything was detailed and labeled."

"When looking for specific details, they were not always easy to find"

"Specific information required more searching"

Undergraduate comments

"If you searched for something is came down in level from the most relevant. Purple highlighted the ones youíve already been to before so could go to blue"

"The links are easy to use. The layout was simple."

Summary of Open-ended survey/Interview question Findings

In sum, the interviews and open-ended survey questions indicate that the majority of participants found the site clear and organized and easy to use and navigate. Some of the interview responses of the 8th and 9th grade students indicate a lack of comprehension of the content of the website and the vocabulary used in the website. This point will be further analyzed and discussed below under question #2.

D. Expert Opinions

Twenty-three experts in the fields of paleoclimate science, geology, and meteorology provided feedback on the CTL web site. This data was primarily collected through email responses to the CTL web site and through one interview. The professionals consisted of scientists, researchers and university professors. Because the majority of respondents were not responding to a survey or interview, not every response addressed all of our research questions.

Overall, the experts agreed that the site was clear, organized, and a useful resource. They were split as to whether the site was easy to use and navigate. The experts were also split regarding the helpfulness of the links. The experts who commented on the ease of use of the site were divided as to whether the site was easy to use and navigate with two respondents asserting that the site was easy to navigate and two respondents asserting that the CTL web site was not easy to navigate. Those who found the site easy to navigate liked the ease of the timeline while those who thought the site was not easy to navigate cited the tutorial as unhelpful. All of the respondents who commented on clarity and organization (n=4) thought that the site was clear and organized. Further, three of the experts that commented on whether the site was a useful resource agreed that it was a useful resource, while one expert claimed that it was not a useful resource. Finally, two experts thought that the links were helpful while three experts thought that the links were not helpful. Those who found the links helpful thought they would be useful for educational purposes and those who thought the links were not helpful cited a broken link in the climate history section and reported that the links were scattered and needed more organization. One expert called for a two sentence review of each link. A full listing of comments from science professionals is included in the appendix.

Question #2: Did the CTL Website increase studentsí understanding of climate science, the relationship between climate and human history, the forces that drive climate variability, and the connection between climate and current events.

Organization of Question #2

Question #2 is divided into three sections. The first section will address 8th, 9th, and 12th grade studentsí understanding of climate concepts. The second section will address undergraduate studentsí understanding of climate concepts. The third section will provide an overview of expertsí opinions of the potential of the website to be used as an educational tool.

Determining appropriate audience

The intended audience for the Climate Time Line was originally undergraduate students. The results of this study confirm that, with further development of learning activities and interactive strategies this proto-type holds strong potential for offering an interdisciplinary approach to climate change, especially for undergraduate, high-school, and middle school audiences. Interviews with undergraduate students included feedback such as: "The CTL increased my depth of understanding. I learned stuff I didnít know before." One middle school science teacher saw potential for using the quiz as a type of treasure hunt for his students to help them gain a wide background in climate-related topics.

See the results from the online surveys at: http://www.ngdc.noaa.gov/cgi-bin/paleo/survey/survey.cgi?survey_name=survey (short survey) and http://www.ngdc.noaa.gov/cgi-bin/paleo/survey/surveylong.cgi?survey_name=surveylong (long survey.)

Limited time to view site

All middle school and high school student participants in the evaluation viewed the site within a limited time frame. 8th grade students spent 30 minutes on the web site, less than a typical science classroom period. Due to technical difficulties with computers, 9th and 12th graders were only able to view the site for 15 minutes (they were given a 30 minute time slot but technical problems resulted in the students having to reload the web page various times). Regardless of time constraints, 75% of 8th grade students were able to raise their quiz scores. Further, the majority of students were able to list at least one fact about climate science and climate history after viewing the site within a limited time frame. The majority of 8th graders were also able to list at least one fact about climate variability and climate and current event. Therefore, we conclude that the web site could effectively be used as a research tool within the time constraints of a classroom period. However, textual and vocabulary modifications are also needed to make the site more comprehensible to 8th, 9th and 12th graders. Undergraduate students were given no time constraints and were not only able to increase quiz scores but reported increased understanding of all climate topics. See the Instructional Design evaluation of CTL at http://www.ngdc.noaa.gov/paleo/ctl/evaluation2.html

Benchmarks of understanding

We utilized benchmarks of understanding provided by Mark McCaffrey of CIRES. We compared these benchmark to studentsí interview/open-ended survey responses to determine whether student responses demonstrated general understanding of a concept, partial understanding or misconception of a concept or no understanding. We used this method for understanding of climate science, the relationship between climate and human history, the forces that drive climate variability, and the connection between climate and current events. Given the extremely limited time constraints and the limited background in the topic of weather and climate variability for many of these students, many students were unable to gain more than a partial understanding. Some misconceptions were also revealed, as when one student said they learned that the sunís obliquity is 23.4, confusing this with the Earthís obliquity of 23.5 degrees.

Section One: 8th, 9th and 12th grade students

  1. Survey Questions and Interview/open-ended survey Findings: 8th, 9th and 12th grade students

All 8th, 9th and 12th grade participants in the Climate Timeline evaluation were asked to answer Likert scale survey questions regarding an increased understanding of climate science, the relationship between climate and human history, the forces that drive climate variability, and the connection between climate and current events. In addition, these students were asked to answer either open-ended questions on the survey or open-ended interview questions regarding their level of understanding of climate science, the relationship between climate and human history, the forces that drive climate variability, and the relationship between climate and current events. The 8th, 9th and 12th grade responses are detailed below.

Likert Scale Survey Statement #1: "The website increased my understanding of Climate Science"

All participants were asked to respond to this statement on a survey. Responses were on a Likert scale ranging from strongly agree, agree, neutral, disagree, to strongly disagree. The 8th, 9th and 12th grade answers are detailed below in table #10.

Table #10. Question #1g. The website increased my understanding of climate science.

The majority of all students "agree" and "strongly agree" that the website increased their understanding of climate science (52% of 8th graders, 43% of 9th graders, and 35% of 12th graders). In order to validate these findings, we followed up on the survey answers with a specific open-ended written question or an open-ended interview question: "What scientific information about climate did you learn from the website."

Open-ended survey/interview question #1: "What scientific information about climate did you learn from the website?"

The benchmark utilized to determine understanding in this category is that students demonstrate an understanding that weather and climate naturally vary due to cycles (such as annual and daily cycles) and climate processes such as ENSO, etc. An example of a student response that demonstrates general understanding is "major volcanic events and how they impact global climate." An example of a student response that demonstrates partial understanding is "I learned there was a major volcanic event in 1259 A.D." The appendix contains more quotes for all categories of learning.

Student answers to the above question varied from listing major climatic events such as volcano eruptions or droughts to stating that they learned "nothing." Categories of student answers are detailed below in table #11 and #12. For a description of each category of answer, see appendix. Categories that represent "general understanding" of the concept are marked with "G." Categories that represent "partial understanding" are marked with "P."

Table #11. What scientific information about climate did you learn from the website.

Student Responses

Student Answer

Percent of 8th grade who answered

Percent of 9th grade who answered

Percent of 12th grade who answered

"Nothing"

39%, n = 9

42%, n = 14

64%, n = 9

G = Climate Changes over time

18%, n = 8

13%, n = 6

14%, n = 2

Ice Age/Glacial Meltdown

0%, n = 0

6%, n = 2

7%, n = 1

P =Earthís different climates affect people

0%, n = 0

9%, n = 3

7%, n = 1

Anazasi People

0%, n = 0

3%, n = 1

0%, n = 0

Statistical Information on Climate Events

17%, n = 4

12%, n = 4

0%, n = 0

P = Major Climate Events

22%, n = 5

15%, n = 5

7%, n = 1

Earthís movement and Climate

17.4%, n = 4

0%, n =0

0%, n = 0

*Note: Some students marked more than one category of answer

Table #12: What did you learn about climate science?" Percent of students who reported "nothing" and percent of students who listed at least one fact.

What did you learn about climate science?

 

18% of 8th graders, 13% of 9th graders, and 14% of 12th graders reported facts from the website that reflected general understanding of climate science. 22% of 8th graders, 24% of 9th graders, and 14% of 12th graders reported facts from the website that reflect their partial understanding of climate science. 34.4% of 8th graders, 21% of 9th graders, and 7% of 12th graders reported accurate facts from the website but these facts did not reflect any type of understanding of climate science.

The percentage of students who responded that they learned "nothing" differed according to grade, although not according to time on computer. 9th, 8th and 12th graders all spent the same amount of time on the computer, yet approximately 40% of both 9th and 8th graders reported that they learned "nothing" from the website. Almost 2/3 of 12th graders (64%) reported that they learned "nothing."

Likert Scale Survey Question #2: The web site increased my understanding of the relationship between climate and human history.

All participants were asked to respond to this statement on a survey. Responses were on a Likert scale ranging from strongly agree, agree, neutral, disagree, to strongly disagree. The 8th, 9th and 12th grade answers are detailed below in table #13.

Table #13: The Website increased my understanding of the relationship between climate and human history

The website increased my understanding of climate and history

The majority of 9th and 12th grade students "agree" that the website increased their understanding of the relationship between climate and human history (55% of 9th graders and 71% of 12th graders). A lesser number of 8th graders "agree" that the website increased their understanding of the relationship between climate and human history (35%) and many of the 8th grade students (39%) took a neutral stance. In order to corroborate these findings, we asked the 8th, 9th and 12th graders a specific open-ended question about their understanding of the relationship between climate and human history,

Open-ended survey/interview question #2: "What historical information about climate did you learn?"

The benchmark used to determine student understanding is that students gained an appreciation that climate factors such as drought can play a role in human historyÖand that much of human activity relates to coming to terms with climatic and hydrologic variability. A student response that demonstrates general understanding of this concept is "Back in history there were droughts and drier, but there were also a lot of floods." Although this response does not refer to the impact of floods on human society, the student has gleaned an understanding that there are cycles of droughts and floods throughout history. A student response that demonstrates partial understanding of a concept is "The Black Sea was the last thing that melted in the ice age."

Student answers to the above question varied from "nothing" to listing statistical information on climatic events such as the dates of floods or droughts. Categories of student answers are detailed below in table #14 and #15. For a description of each category of answer, see appendix. Categories that represent "general understanding" of the concept are marked with "G." Categories that represent "partial understanding" are marked with "P."

Table #14. "What did you learn about climate history?"

Student Responses

Student Answer

Percent of 8th graders who answered

Percent of 9th graders who answered

Percent of 12th graders who answered

Nothing

26%, n = 6

51%, n = 17

64%, n = 9

P =Climate Changes over time

21.7%, n = 5

15%, n = 5

14.2%, n = 2

Ice Age/Glacial Meltdown

9%, n = 3

13%, n = 3

7%, n = 1

G =Climate affects people and people affect climate

0%, n =0

3%, n = 1

7%, n = 1

Anasazi People

3%, n = 1

4%, n = 1

0%

Patterns Among the Chaos

0%, n = 0

0%, n = 0

14%, n = 2

G =Statistical Information on Climatic Events

39%, n = 9

21%, n = 7

0%, n = 0

Note: Some students listed more than one category of answer.

Table #15: "What did you learn about climate history?" Percent of students who answered "nothing" and percent who listed at least one fact.

What did you learn about climate history?

39% of 8th graders, 24% of 9th graders, and 7% of 12th graders reported facts that reflected their general understanding of the relationship between climate and human history. In addition, 21.7% of 8th graders, 15% of 9th graders, and 14.2% of 12th graders reported facts that reflected partial understanding of the relationship between climate and human history. 12% of 8th graders, 17% of 9th graders and 21% of 12th graders reported accurate facts from the website but these facts did not necessarily reflect an understanding of the relationship between climate and human history.

While the majority of 8th, 9th and 12th graders agreed on the survey that the web site increased their understanding of the relationship between climate and human history (35% of 8th graders, 55% of 9th graders, 51% of 12th graders), a majority of 9th graders (51%) and 12th graders (64%) responded that they learned "nothing" from the web site when asked to describe their learning in open-ended questions or interviews. Only 26% of 8th graders responded that they learned "nothing" about climate history in the interview.

Likert Scale Survey Question #3: This web site increased my understanding of the forces that drive climate variability.

All participants were asked to respond to this statement on a survey. Responses were on a Likert scale ranging from strongly agree, agree, neutral, disagree, to strongly disagree. The 8th, 9th and 12th grade answers are detailed below in table #16.

Table #16: The website increased my understanding of the forces that drive climate variability

The site increased my understanding

In table #16, a majority of 8th grade and 9th grade students reported a "neutral" stance as to whether the website increased their understanding of the forces that drive climate variability (44% of 8th graders respond "neutral" 47% of 9th graders respond "neutral"). The majority of 12th graders, however, "agree" that the website increased their understanding of the forces that drive climate variability (29% "strongly agree" and 36% "agree"). .In order to corroborate these responses, we asked the students an open-ended question concerning their understanding of the forces that drive climate variability.

Open-ended survey/interview question #3: "What kinds of forces drive climate variability?"

The benchmark used to determine whether students understood about climate variability is whether students learned about different time scales---daily (Earthís rotation), annual (Earthís orbit and tilt of axis causing season), decadal (ENSO), and long term (orbital cycles) as well as climatic events such as volcanoes. An example of a student response that demonstrated general understanding of this concept is ""The way the earth is tilted" referring to the tilt of the earth causing seasonal change. An example of a student response that demonstrated partial understanding is "water, sun" which does not fully meet the benchmarks set for learning. The appendix contains more student responses to this question.

Student answers to the above question ranged from "nothing" to listing the human impacts on climate variability such as pollution and greenhouse gasses. Categories of student answers are detailed below in table #17 and #18. For a description of each category of answer, see appendix. Categories that represent "general understanding" of the concept are marked with "G." Categories that represent "partial understanding" are marked with "P."

Table 17: What kinds of forces drive climate variability?

Student Responses

Student Answer

Percent of 8th graders who answered

Percent of 9th graders who answered

Percent of 12th graders who answered

Nothing

22%, n = 5

82%, n = 27

71%, n = 10

G =Major Climatic Events

22%, n = 5

0%, n = 0

14%, n =2

Human impacts (pollution, greenhouse gasses)

39%, n = 9

3%, n = 1

28.5%, n = 4

G =Earthís pull and tilt

13%, n = 3

6%, n = 2

7%, n = 1

The sun and the moon

13%, n = 3

9%, n = 3

0%, n = 0

P =Wind, Water, and Precipitation

4%, n = 1

6%, n = 2

0%, n = 0

Note: Some students listed more than once category of answer.

Table #18: "What kinds of forces drive climate variability?" Percent of students who reported "nothing" and percent of students who listed at least one fact.

What kinds of forces drive climate variability?

78% of 8th graders, 18% of 9th graders, and 29% of 12th graders were able to articulate at least one fact from the website. Of these students, 35% of 8th graders, 6% of 9th graders, and 21% of 12th graders reported facts which reflected their general understanding of the forces that drive climate variability. In addition, 4% of 8th graders and 6% of 9th graders reported facts which reflect their partial understanding of the topic

While the majority of 8th and 9th graders report on the survey that they are "neutral" on whether or not the web site increased their understanding of the forces driving climate variability, the great majority of 9th graders (82%) responded that they learned "nothing" from the web site when asked in an interview. Only 22% of 8th graders reported that they learned "nothing" about climate variability when interviewed. While the majority of 12th graders reported that they "agree" and "strongly agree" that the website increased their understanding of the forces that drive climate variability(65%), the majority of 12th graders (71%) also report that they learned "nothing."

Likert Scale Survey Question #4: This web site increased my understanding of the relationship between climate and current events.

All participants were asked to respond to this statement on a survey. Responses were on a Likert scale ranging from strongly agree, agree, neutral, disagree, to strongly disagree. The 8th, 9th and 12th grade answers are detailed below in table #19.

Table #19: The website increased my understanding of the relationship between climate and current events

The site increased my understanding of climate and current events

The majority of 8th and 9th grade students "agree" that the website increased their understanding of the relationship between climate and current events (47% of 8th graders, 42% of 9th graders). The 12th graders are equally divided between "strongly agreeing" that the website increased their understanding (36%) and "neutral" on whether the website increased their understanding (36%). Again, we asked an open-ended question to determine exactly what the students had understood about the relationship between climate and current events.

Open-ended survey/interview question #4: "What did you learn about the relationship between climate and current events?"

The benchmark used to determine understanding about the relationship between climate and current events is whether students understood global warming, ENSO, and major climatic events such as droughts. An example of a student response that demonstrates general understanding is "Greenhouse gasses cause global warming." An example of a student response that demonstrates partial understanding is "climate controls a lot of our everyday events." This student acknowledges that there is a connection between climate and current events yet does not specify what climate events are currently affecting us or how they are affecting us. The appendix contains a full listing of student responses to this question.

Student answers to the above question ranged from "nothing" to listing the human impacts on climate variability such as pollution and greenhouse gasses. Categories of student answers are detailed below in table #20 and #21. For a description of each answer category, see appendix. Categories that represent general understanding" of the concept are marked with "G." Categories that represent "partial understanding" are marked with "P."

Table #20: What did you learn about the relationship between climate and current events?

Student Responses

Student Answer

Percent of 8th graders who answered

Percent of 9th graders who answered

Percent of 12th graders who answered

Nothing

52%, n = 12

72.7%, n = 24

78.5%, n = 11

G = Humans affect climate

4%, n = 1

6%, n = 2

7%, n = 1

G=Climate affects Humans

12%, n = 3

13%, n = 4

7%, n = 1

G =Statistical Information on Climatic Events

13%, n = 3

7%, n = 1

3%, n = 1

P = Relationship between past and present

17.3%, n = 4

3%, n = 1

0%, n =0

Note: Some students listed more than one category of answer.

Table #21: "What did you learn about the relationship between climate and current events? Percent of students who reported "nothing" and percent of students who listed at least one fact.


46.3% of 8th graders, 27.3% of 9th graders, and 17% of 12th graders articulated at least one fact from the website pertaining to climate and current events. 29% of 8th graders, 26% of 9th graders, and 17% of 12th graders report facts that reflect their general understanding of the relationship between climate and current events. In addition, 17.3% of 8th graders and 3% of 9th graders report facts that reflect their partial understanding of the topic.

While the majority of 8th, 9th and 12th graders "agreed" or took a neutral stance on the survey that the web site increased their understanding of the relationship between climate and current events, the majority of students in all grades responded that they learned "nothing" from the website about the relationship between climate and current events when asked in an interview or open-ended survey question (52% of 8th graders, 72.7% of 9th graders, and 78.5% of 12th graders).

Survey Question #5: Was any aspect of the website unclear?

In addition to the above questions on a Likert scale asking about studentsí understanding of specific topics of the website, the 8th, 9th, and 12th grade students were also asked a general question about whether any aspects of the website were unclear. Their answers are detailed below in table #22.

Table #22. Was any aspect of the Website Unclear?

The majority of 8th grade students (61%) found aspects of the website unclear. The majority of 9th and 12th grade students report that they did not find any aspect of the website unclear (59.3% of 9th graders, 80% of 12th grades). We then asked the 8th, 9th, and 12th graders in interviews to indicate which aspects of the website specifically they found unclear.

Open-ended survey/interview question #5: "Which aspects of the website did you find unclear?"

61% of the 8th graders reported on the survey that they found aspects of the website unclear. In interviews, the 8th grade students reported a lack of understanding of the science content and the level of vocabulary used in the website:

"The language was a little higher than I am used to. It was a lot of new stuff, I didnít have the background to understand it."

"The content was clear. I understand what it was saying but didnít understand why the information was important."

"Once you start reading you know whatís offered, what itís about"

40.7% of the 9th graders also reported aspects of the website unclear on the survey. In interviews the 9th graders reported difficulty in understanding the level of vocabulary and some aspects of the content.

"It could have been worded differently in some parts"

"I didnít understand the graphs"

"I didnít know where to go to get a lot of the information"

"There was too much information"

"If I had known some of the information before [using] the website, [it] would have been very helpful."

While only 20% of the 12th graders reported that aspects of the website were unclear on the survey, in interviews these students also reported difficulty understanding the level of vocabulary and the content of the website.

"It was damn esoteric! The language is a result of the topic being a science."

"The aspect of the PDO and NAO were unclear because I have never heard of them"

Summary of survey, interview and open-ended questions with 8th grade, 9th grade, and 12th grade students.

The data for question #2, presented us with some interesting findings. Survey responses on the Likert scale indicate that the majority of 9t, and 12th grade students report an increase in their understanding of climate science, climate and human history, the forces that drive climate variability, and the relationship between climate and current events. 8th graders reported the least amount of increased understanding for all topics. On the other hand, open-ended survey responses and interview responses indicate that the majority of 9th and 12th grade students report that they learned "nothing" about climate science, climate and human history, the forces that drive climate variability, and the relationship between climate and current events. The majority of 8th graders, however, were able to list at least one fact from the website pertaining to the topic. Of the 8th, 9th, and 12th grade student participants who did indicate specific facts learned from the website, 8th graders were more likely to articulate a fact that reflected their full understanding of the topic than were the 9th and 12th graders.

The last survey question asked of the students, "Did you find any aspect of the website unclear?" indicates that the majority of 8th graders and a minority of 9th and 12th grade students found aspects of the website to be unclear and all students stressed that the vocabulary and science content was too advanced for them to comprehend.

In conclusion, the 9th and 12th grade students appear to perceive an increase in their understanding of climate concepts but when asked to articulate that understanding, the majority of 9th and 12th grade students were unable to do so and reported learning "nothing." On the other hand, the majority of 8th grade students did not perceive an increase of their understanding of climate concepts but the majority of these students were able to articulate at least one fact pertaining to the topic, and often, these facts reflected full understanding of the topic.

In part, these findings can be attributed to the technological problems the 9th and 12th graders experienced with their school computers; problems which the 8th graders did not experience. The 8th graders were able to dedicate the whole of their 30 minute time allotment to researching the website while 9th and 12th graders had to spend time reloading the website when the school computers crashed. Therefore, while all students were given equal time to explore the website, time actually spent exploring the website was less for the 9th and 12th graders than for the 8th graders. At the same time, regardless of time spent exploring the website, all students reported that the websiteís vocabulary and content was too advanced for them to comprehend.

B. Quiz Findings

All participants in the Climate Timeline evaluation took a pre quiz prior to visiting the website (8th graders visited the website for 30 minutes, 9th graders visited the website for 15 minutes, 12th graders visited the website for 30 minutes. The pre and post quiz consisted of questions designed to measure the participants understanding of climate science, climate and human history, the relationship between climate and current events, and the forces that drive climate variability. Unfortunately, technological problems with the school computers caused the 9th and 12th grade quizzes to give back inaccurate information and, therefore, the quizzes from the 9th and 12th graders will not be used in this evaluation. The 8th graders were given paper and pencil quizzes.

8th grade quizzes

Below in table #23 is a breakdown of 8th grade student pre and post quiz scores. All 8th graders visited the website for 30 minutes. 75% of the 8th graders increased their score on the quiz after visiting the website. The average score of all 8th graders on the pre-test was 6 correct out of 19 questions, or 31.5%. The average score of all 8th graders on the post-test was 8.5 correct out of 19 questions, or 44.7%.

Table #23. Breakdown of 8th grade quiz scores

Student Respones

Change between pre and post quiz

# of students

Percentage of test-takers

Score increased

18

75%

Score stayed the same

4

16.6%

Score decreased

2

8.3%

Question #1 "What did you learn about climate science?":

The 8th grade quiz scores reflect that the 8th graders were able to successfully locate about half of the quiz answers which pertained to climate science in the allotted 30 minute time period. This finding indicates that the web site may be a useful classroom tool for research or for looking up information on climate science.

Question #2: "What did you learn about climate and human history?":

The 8th grade quiz scores also reflect that they were able to successfully locate enough history answers to increase the history portion of their quiz scores an average of 22%. This is significantly more than the 6.6% increase in 8th grade scores in the climate science portion of the quiz. This indicates that the web site may be a useful classroom tool for research or looking up information on climate and human history.

Question #3: "What kind of forces drive climate variability?":

Although the quiz did not directly address climate variability as thoroughly as climate science or history, 8th grade quiz responses indicate that 8th graders were unable to effectively research climate variability on the web site.

Question #4: What is the relationship between climate and current events?"

The 8th grade quiz scores also indicate that almost half of them were able to locate information on the 1997-1998 El Nino event in a 30-minute time period. This may indicate that the web site could be a useful classroom tool for research or locating information on climate and current events.

Summary of 8th grade Quiz scores:

The multiple choice format of the quiz allows us to determine whether students were able to locate answers on the web site correctly in a given time period. The data indicate that 8th graders were successful in finding information on climate science, history, and current events, but were not able to locate information on the forces that drive climate variability. The 8th graders raised their scores by an average of 13% and with an average post-test score of under 50%. Although, they were able to research some topics well, such as El Nino, they were unable to find at least half of the quiz answers in the 30 minute time period. Overall, the data shows that the web site may be an effective research tool for 8th graders to find information about climate science, history, and current events.


Section Two: Undergraduate Students

A. Survey Questions and Interview/open-ended survey Findings: Undergraduate Students

16 undergraduate students filled out surveys in which they were asked about their understanding of climate science, climate and human history, the forces that drive climate variability, and the relationship between climate and current events. Two of these students were also interviewed and asked open-ended questions about their level of understanding. All undergraduate responses are detailed below in tables 24 Ė27.

Survey Question #1: The website increased my understanding of climate science.

All undergraduate students were asked to respond to this statement using a Likert scale ranging from strongly agree, agree, neutral, disagree to strongly disagree. The undergraduate answers are detailed below in table # 24.

Table 24 . The website increased my understanding of climate science.

Survey data indicates that the great majority of undergraduate students (62.5%) strongly agree and agree that the website increased their understanding of climate science.

Open-ended interview question #1: How did the site increase your understanding of climate science?

Two undergraduate students reported that the web site increased their understanding of climate science in an open-ended interview. Below are representative quotes:

"I learned about ENSO, PDO and learned the difference between climate and weather. I also learned about ice ages."

[The climate science content] "was interesting and easy to understand. It connected to my Environmental Sciences major. I wouldíve wanted to use this in one of my Environmental Sciences courses."

Survey Question #2: The website increased my understanding of climate and human history.

All undergraduates were asked to respond to this statement on a likert scale. Their responses are detailed below in table # 25.

Table # 25. The website increased my understanding of climate and human history

Survey data indicates that the majority of undergraduate students (50%) agree that the website increased their understanding of climate and human history.

Open-ended interview question #2: How did the site increase your understanding of climate and human history?

Two undergraduate students reported in an open-ended interview that the website increased their understanding of climate and human history. Below are representative quotes:

"It increased my depth of understanding. I learned stuff I didnít know before."

[The history content] "was interesting. It was very detailed. The timelines got the point across."

"The powers of 10 model helped me understand what has changed"

Survey question #3: The website increased my understanding of the forces that drive climate variability.

Undergraduate students were asked to respond to this statement on a Likert scale. Responses are detailed below in table # 26.

Table # 26. The website increased my understanding of the forces that drive climate variability.

Survey data indicates that the great majority of undergraduate students (68.7%) agree that the website increased their understanding of the forces that drive climate variability.

Open-ended interview question #3: How did the site increase your understanding of the forces that contribute to climate variability?

Two undergraduate students reported that the website increased their understanding of the forces that contribute to climate variability in an open-ended interview. Below are representative quotes:

"There were helpful links to different variabilities"

"It talked about the tilt of the earth, eccentricity, and the human impact."

Survey Question #4: The website increased my understanding of the relationship between climate and current events.

All undergraduate students were asked to respond to this statement on a Likert scale. Responses are detailed below in table # 27.

Table #27. The website increased my understanding of the relationship between climate and current events.

Survey data indicates that the great majority of undergraduate students (75%) agree that the website increased their understanding of climate and current events.

Open-ended interview question #4: How did the site increase your understanding of climate and current events?

Two undergraduate students reported that the website increased their understanding of climate and current events in an open-ended interview. Below are representative quotes:

"It had a lot of timelines that say what happened since 1980. I can look at the cloud cover in Africa right now."

"Yes, it helped explain global warming but there is more about history than current events."

Summary of survey, interview findings: undergraduate

Both survey responses and interview data demonstrate that the website increased undergraduate student understanding of climate science, climate and human history, the forces that drive climate variability, and climate and current events. The majority of undergraduate students who completed the survey reported that they strongly agreed or agreed that the website increased their understanding of the four areas. The two undergraduate students who were interviewed also reported increased understanding of all four areas. Data therefore indicates that the climate timeline tool is an appropriate website for the undergraduate level; the undergraduate students report increased understanding of climate topics after visiting the website, their quiz scores increased after visiting the website, and as reported in section #1, undergraduate students report ease of use when manipulating the website. The website could be effectively utilized as a research or learning tool for undergraduate students.

B. Quiz Scores: Undergraduate

Below in table 28 we present a breakdown of the undergraduate student quiz scores. Undergraduate students visited the website for a varied amount of time ranging from 11 minutes to 2 hours. Regardless of time spent on the website, 100% of undergraduate students increased their quiz scores after visiting the website. Undergraduates increased their scores by an average of 42%.

Table #28. Breakdown of undergrad quiz scores

Student Results

Time spent on computer

# of students

Pre-test average score

Post-test average score

Average increase in score

11-30 minutes

7

37%

77%

40%

30-59 minutes

8

35%

81%

46%

1hr +

0

n/a

n/a

n/a

2hr+

3

37%

85%

48%

Because the undergraduates submitted their quizzes on-line, we have data indicating an increase or decrease of score, but are not able to disaggregate the data by question.

Summary of undergraduate quiz scores

The undergraduate data show that 100% of the 18 students who took both the pre-test and post-test were able to raise their scores. They raised their scores an average of 42% which is much higher than the 13% increase of the 8th grade students. This indicates that the web site is a more appropriate research tool for undergraduate students than for 8th graders. Undergraduates were more successful in locating correct answers and increasing their scores. Although most of the undergraduates had more time on the computer than the 8th graders, their increases did not vary significantly by time spent on the computer. Those who spent less than 30 minutes on the computer were still able to raise their scores by 40%. Therefore, the web site could be an effective research tool for undergraduates. In fact, one undergraduate stated in an interview, "I would definitely use this site for research. It is much better than other climate sites Iíve visited like the ones at NCAR." Based on quiz scores, the web site is an appropriate research tool for undergraduate students.

Section Three: Expert opinions.

Twenty-three experts in the fields of paleoclimate science, geology, and meteorology provided feedback on the CTL web site. This data was primarily collected through email responses to the CTL web site and through one interview. The professionals consisted of scientists, researchers and university professors. Because the majority of respondents were not responding to a survey or interview, not every response addressed all of our research questions.

The experts identified several areas of the CTL web site that needed additional information or more thorough coverage. They also identified several ways that the web site can be improved to facilitate classroom learning. Seven of the twenty-three experts identified areas that need additional information. These areas include a more detailed glossary, more detail in major glaciations in Earthís history, more information about continent formation, tectonic activity, and a comparison of past paleo observation methods with modern space-based measurements. Three experts thought that the coverage of global warming should place greater emphasis on the 20th century surface temperature record and research that demonstrates that the anthropogenic origins of accelerated warming since the 1970s. Two experts also thought that the Black Sea evidence on the web site needed to be reconsidered. Finally, two experts identified ways to enhance classroom learning with the web site such as the inclusion of on-line classroom exercises on the site, the inclusion of interactive animation such as Macromedia Flash, and an attendance to different learning styles such as kinesthetic, auditory and visual learners. A full listing of comments from science professionals is included in the appendix.

In addition to the experts listed above, a 6th-8th grade science teacher visited the website and was interviewed afterwards. This teacher suggested that the content of the website is more suitable for high school age and above. However, he stated that the tabular structure in the overview and history pages work well for middle school students to gleam information for the different time periods. He suggested that if information were presented in small chunks, like in a bulleted format rather than the paragraph format, it would be easier for middle school students to pull out information. He also suggested that some of the dates in the overview or history sections be linked to either the science pages or data access pages in order to explain possible climate patterns that may have influenced a particular event, like ENSO or other climate cycles or events. This teacher reported that the quiz we gave the 8th grade students was quite worthwhile in that it set up a "scavenger hunt" format to retrieve information from the web site. He particularly liked the fact that the quiz questions encourages students to explore different places in the website, covering different themes. This teacher reported that the CTL would make a good resource for teachers in prepping for Earth Science classes.

Return to CTL Evaluation top
Return to CTL Homepage