Data Challenges
This past week I was lucky enough to work with data specialist Jennifer Borgioli to discuss possible data collection measures to measure the goals established for this newly established MST position.
MST Goals:
- Increase student opportunities to explore long-term science & math investigation
- Increase student performance in both science content and in using the scientific method
- Increase teacher confidence with technology integration
Since this is the first year I don’t have my own classroom, there are several challenges I’m working through. One is the difficulty in measuring an increase in student performance. Most of the time this year, students have visited the MST room for a series of 3-5 lessons as part of a larger unit in science. Recently for 5th grade, the purpose of the lessons is to build background for a unit on the changing Earth (geology). I have data on formative measures which show some growth, but not as significant as what would be seen later in a unit. Since I’m not teaching an entire unit, nor am I working with all classrooms in the school, what is the best way to measure the increase in student performance?
The other piece to this is in regards to project and problem based learning experiences. Part of my focus this year has been directed at implementing an HP team grant with three classroom teachers (2nd, 3rd, and 4th grade). We have students investigating water quality at local ponds/stream through the year using various data collection measures. They have worked liked scientist collecting and analyzing data, developed their own inquiry questions, have had the opportunity to explore a topic in-depth, worked in collaborative settings, and will be presenting their findings to an authentic audience (Monroe County Water Coalition Meeting in March at the Brighton Town Hall). The way this learning experience is assessed is much different than the format of the NYS 4th grade science test. What happens if the group of 4th graders on this project does not perform as well on the NYS test as others in the grade level but has demonstrated a deeper understanding of living environment concepts through their experiences and tasks in the classroom?
One idea she shared by Jennifer was to interview a sample of students to gather information on their perceptions of learning. Examples: How well do you feel you learned about ___ through this project/problem? How do you feel this experience has helped you? How involved in learning do you feel you were during this project/problem?
From what I’ve researched on my own, the format of this instrument is intended to elicit students’ estimates of how much they gained from the class, and to relate their gains to particular aspects of the class pedagogy. Its particular focus is on what students gained from the class rather than asking students what they “liked” or “valued” about their classes.
I’m interested to hear more about how this type of data collection can be used to measure the identified goals. How can the questions be tweaked to elicit the most meaningful responses? How many students should be surveyed and how should the students be chosen? Also, what other less common forms of data collection can be used?
Now that I’ve had the opportunity to work in this position for 6 months, there are other unintended outcomes, not directly related to student performance, but rather to an increase in teacher confidence and use of student-centered practices such as inquiry and PBL. Should data be collected on these unintended goals? If so, what data collection measures would work best? Ahh…the challenges of learning how using data effectively!
on February 9th, 2008 at 7:57 pm
Sometimes the demands of the data-collection world can blind us to the forest as we work to document the trees. One type of “data collection” that I would urge you to consider is narrative. In a narrative form, the data would indicate the presence of specific student behaviors and attitudes in contrast to their absence in classes that have not been involved in your project. For example, the fact that the students are making a presentation before the Monroe County Water Coalition is a strong piece of data that (I presume) the other (control?) groups cannot evidence. Video and photo documentation of the excitement of your learners can be contrasted to the status quo attitudes of students not involved in your program.
I just fear that if we let demands for data collection determine what’s most valuable, we risk losing track of the best outcomes that are most important.
If you think it would be helpful, I would be happy to provide some narrative description of the positive outcomes I’ve seen from my end.
JS
on February 10th, 2008 at 1:49 pm
Jim,
Thanks for the suggestion. You hit the nail on the head when you wrote, “…if we let demands for data collection determine what’s most valuable, we risk losing track of the best outcomes that are most important.” This is my biggest concern this year in this newly created role. I also feel that substantial gains, in student achievement or otherwise, may definitely require more than 10 months to be achieved.
I’d love to see some examples of narrative descriptions of outcomes and the protocols for this type of data collection measure. Is this something your students and I could discuss together?
on February 12th, 2008 at 1:10 pm
I think there can be many valid ways of collecting data. Certainly anedotal data or attitudes toward inquiry or science can also be data as well as highlighting how many real world problems children have responded to this year compared with last year.
I am interested in how we might measure the increase in teacher confidence also in designing inquiry units.
The
on February 13th, 2008 at 9:50 pm
Wendy: I really enjoyed our conversation last week and learning about your lab and the exciting work that you are doing with your students. I agree that a narrative can be a powerful way to capture data that might be otherwise overlooked. At the same time, generally speaking, a narrative still needs to be hung or focused around a guiding question or focus. Often times, these guiding questions can be found within program or school goals. If we look back at the first goal that you established: Increase student opportunities to explore long-term science & math investigation, there are several questions that might reveal interesting data. For example, for students who do not participate in the MST lab, how many opportunities do they have to explore long-term science and math investigations? (once a month? Once a quarter?) How long is long-term? (Is it a week? A month? Until it’s done?)
Similar questions could be asked about the other goals. I’m especially interested in second goal and the issue of the scientific method. Is it possible to ask students to self-asses their understanding of the different components of the scientific method?
on February 18th, 2008 at 11:01 am
Wendy- This site of yours is great. I really appreciate your depth of thinking and quality of authentic learning experiences you provide students and teachers. It will be interesting to look at data regarding students who did not participate, or participated infrequently in MST, versus those who frequented your classroom. In addition, some sort of survey for teachers will prove quite valuable, both to determine how MST impacted their own, and student learning. My opinion is the depth of understanding and worldly applications of concepts is a priority- classroom teachers are responsible then to follow up on your lesson and make connections that will better prepare students for state tests, etc.
on February 19th, 2008 at 12:49 am
Jennifer,
Thanks for your comments. I’m in the process of working on a scientific method task that could be used as an assessment. My thought was to present students with a problem or perhaps inconclusive data and see how they would go about solving it. They would need to be able to transfer the scientific process that they’ve experienced in class to a new situation. Do you have any suggestions?
on February 19th, 2008 at 10:00 am
Wendy,
Sometimes the process of posing the problem for investigation is an overlooked part of the scientific method. When we give children the problem to investigate, we are doing a lot of the messy work for them. A skill that I think you might want to include in your assessment is the skill of problem posing. At a low level you could present a number of questions and ask the children to identify which ones lend themselves to scientific investigation. At a higher level you might present a scenario and ask them to generate questions that could be investigated using the scientific method. If you pose it as an open-ended question (How many different scientifically-investigatable questions can you think of?), then your scoring could be two dimensional: You could evaluate the legitimacy of the questions and you could simply count the quantity of questions generated.