NERRS banner

Inventory and Assessment of K-12 and Professional Teacher Development Programs in the National Estuarine Research Reserve System

C. INTRODUCTION

Methodology

In October 2002, the Formal Education Workgroup and members of Pandion Systems’ team had an initial telephone meeting to begin development of the project. Following the meeting, Pandion Staff met with Education Coordinators (ECs) at their annual meeting in Cape Cod at the Waquoit Bay Reserve. At this meeting the formal objectives and data collection topics were discussed so that the Pandion team could confirm the objectives, refine a work plan, and establish deliverables for the project. The data collection methods agreed upon by the group were online surveys, in-depth telephone interviews, and limited site visits.

Data Collection
Pandion administered three on-line surveys to various NERRS staff members. The ECs answered two online surveys: a general survey that collected data on their entire K-12 and PTD program and an individual program survey that collected specific data on each current K-12 and PTD program. The Reserve Managers (RMs), Research Coordinators (RCs), and Stewardship Coordinators (SCs) answered a survey that collected data about communication at each Reserve and about support for K-12 and PTD education programs.

The first online survey was a general survey that targeted the ECs. It was designed to provide initial data to address the four project objectives. At the end of the general survey each EC was asked to list each current K-12 program and current PTD program (taught within the last year).

The second online survey was designed to capture information about each individual K-12 and PTD program that was listed in the general survey. ECs submitted a separate survey for each program. These questions were used to satisfy the first objective of the project: to provide a system-wide inventory of the NERRS K-12 and PTD education programs.

Following the two initial online surveys, each EC was interviewed in an in-depth telephone survey that included several follow up questions. The open-ended questions were designed to clarify answers given in online surveys and to seek in-depth information for specific project objectives.

The third online survey was sent to Reserve Managers (RMs), Research Coordinators (RCs), and Stewardship Coordinators (SCs). These questions were designed to capture data to determine how research and stewardship aspects of the Reserves contribute to K-12 and PTD programs.

Finally, site visits were conducted, spending one day each at four Reserves, asking the same questions from the in-depth telephone interview. Site visits allowed us to spend more time discussing some of these questions. In addition, we conducted brief face-to-face interviews with other staff members including RMs, RCs, and SCs. We also viewed and gathered some samples of K-12 and PTD materials.

Of the 25 existing Reserves, 24 responded to the online surveys (the 25th Reserve does not have an EC), and 24 also responded to both the individual program online surveys, and the telephone interview (N=20) or site visit interview (N=4). We also collected online surveys from 19 RMs, 17 RCs, and 9 SCs. To protect respondent confidentiality each survey was coded and access was restricted to the study team.

Data Interpretation Constraints
A multiple-choice survey with Likert scale questions was used to determine responses for many categories. When this type of data is gathered there are constraints based on the choices available for respondents to answer. During the follow-up interviews, ECs were more flexible when they could explain their answers. An example occurred in the follow-up interview when each EC was asked about their feelings for developing a national program. Some ECs that indicated in the online survey that they did not want a national program were able to explain why and add clarifiers that identified the type of national program or activity that would work for them.

There are two other cases where there was confusion about a question and the study team found that there were multiple interpretations of the question.

The first case involves confusion about the definition of formal evaluation. The online individual survey posed the question “What type of formal evaluation has been done for this program?” with a list of corresponding answers to choose from. Another question in the follow-up telephone interview asked “For programs that have been formally evaluated, what were the results?” In both questions, ECs could have interpreted the definition of formal evaluation as a program that has had specific data collected and analyzed quantitatively; or as a program that has feedback collected and has been reviewed on a case by case basis, but has not had any formal quantitative results tabulated. In some cases ECs also interpreted the definition of formal evaluation as an evaluation conducted by an outside party.

The second case involves a question from the online general EC survey, “Rate the degree to which your Reserve educational programs (K-12 and PTD) are different from other coastal related programs.” There were 10 parts where ECs were asked to rate on a scale of 1 to 10 (where 1 equals not at all different and 10 equals very different). When analyzing the results there was a double meaning that made the results unclear. The question was interpreted differently by ECs. For example, one part asked to rate on a 1-10 scale (where 1 equaled not at all different and 10 equaled very different) “We integrate research from the Reserve into programs.” If a 5 rating was given it was unclear if they did not integrate research into their programs thus there is no differentiating factor or if they did integrate research and the other providers also did too. In retrospect this question should have been separated into two parts.