Executive Summary for the LEAD Center's evaluation of the CRPC's
Spend a Summer with a Scientist (SaS) program
Baine Alexander and Julie Foertsch
The LEAD Center at the University of Wisconsin-Madison
Full version available at http://www.cae.wisc.edu/~lead/pages/products/sas.pdf
copyright: The LEAD Center
One of the most vexing problems in higher education has been the dramatic underrepresentation of African Americans, Hispanics, and American Indians among advanced degree holders, especially in the areas of science, mathematics, engineering, and technology. Numerous programs at both the national and institutional level have attempted to address this problem with minimal success (Manger, 1996; Manzo, 1994; Massey, 1992; Phillip, 1993). Despite two decades of effort, it remains the case that few minority students enroll in graduate programs in these areas, and even fewer go on to complete their advanced degrees (Manger, 1996; Massey, 1992).
The Spend a Summer with a Scientist (SaS) program sponsored by the Center for Research in Parallel Computation (CRPC) at Rice University is a program that addresses both the recruitment and the retention of minority graduate students, particularly those in applied mathematics, the computational sciences, and technology. The SaS Program was created in 1989 by Dr. Richard Tapia, a nationally-esteemed professor of applied mathematics and the CRPC's Director of Education and Human Resources, who administers the program with the help of senior graduate student participants. The ultimate goal of the SaS program is to increase the number of ethnic minorities and women in mathematics, the computational sciences, and technology by giving them the support and sense of community needed to succeed in graduate school. Undergraduates and graduate students are given summer financial support to participate in a research project under a mentor/advisor who is either a faculty member or graduate student at Rice University. The topic, degree of difficulty, and amount of structure associated with this project are adjusted to suit each student's needs, whether they are a graduate student completing dissertation research or an underclassman from a small college who has never done research before. Students who need to catch up on their requirements or prepare for upcoming courses may also have a program tailored to meet that need. Participants generally give an accounting of their activities at the end of the summer, either through a written report or a presentation.
There is a strong community aspect to the Summer with a Scientist program. Participants from off-campus are recruited and given practical assistance by the Rice graduate students already in the program, and participants see each other often during the course of the summer, both formally and informally. The entire group of SaS participants attends a weekly seminar with Dr. Tapia to discuss issues of research and professional development, race and gender, and problems or issues that have come up during the week. This weekly get-together is an opportunity to get to know and respect the other participants and to learn important lessons about what it means to be an academic, a researcher, and a minority in one's field. The program also includes an outreach component: participants serve as mentors to two groups of high school students for two weeks during the summer, and they give presentations of their research to K-12 teachers who attend a CRPC-sponsored summer workshop.
2. The Goals and Methodology of the LEAD Evaluation
In the summer of 1997, the LEAD Center was commissioned by the National Partnership for Advanced Computational Infrastructure (NPACI) to evaluate the Summer with a Scientist program. The primary interest of the client for the SaS evaluation, Dr. Richard Tapia, was to determine whether the SaS program could be replicated at other institutions so that the impact of this presumably successful minority recruitment and retention program could be broadened to a national scale. To answer this question, it is first necessary to determine whether the program is indeed successful at achieving its ultimate goal of increasing the number of women and minorities who enroll in and complete graduate school in mathematics, the computational sciences, engineering or technology. Next, one must determine which program elements are essential to its successful outcomes. If the essential elements are not found at, or cannot readily be established at other universities, it suggests that the program will be difficult to replicate elsewhere.
To understand the impact of the SaS program on its participants, the LEAD research team collected data from a number of different sources using both qualitative and quantitative research methods. In July of 1997, two LEAD researchers traveled to Rice University to observe the 1997 SaS program in progress, to interview current participants individually and in two focus groups, and to interview program administrators and staff about their goals and experiences with the program. These research activities provided a base of information about the program and allowed the interview protocols for student participants to be piloted and refined.
To answer the question of whether the program is successful at its primary goal of encouraging minority and female participants to enroll in graduate school and complete their graduate degrees, it was necessary to track previous participants in the program to see what they had done in the years since leaving the program. Beginning in October of 1997 and continuing through April of 1998, researchers at the LEAD Center worked with SaS staff members to obtain outcome information on 52 of the 68 former and current SaS participants since 1991. This tracking information was used to determine the basic program outcomes pertaining to graduate degree pursuit and attainment.
Of course, even if a high percentage of SaS participants go on to enroll in graduate school and/or obtain graduate degrees, these satisfactory outcomes cannot be ascribed to the influence of the SaS program alone without further research. It could be that many of the program participants would have enrolled in graduate school and persisted long enough to obtain their degrees whether they had participated in the program or not. To determine the impact of the program itself, it is necessary to do one of two things: (1) find a comparatively-sized group of minority students at the same or a very similar institution who had not participated in the program and compare their graduate school enrollment and degree attainment to that of SaS participants; or (2) interview and survey SaS program participants about the impact of the program on their beliefs and attitudes about graduate school, their motivation to pursue graduate degrees, and their overall ability to do so. The first option could not be pursued in this case, since all minority graduate students in the computational sciences at Rice are already enrolled in the SaS program and Rice's environment in unique enough and its number of minority graduate students high enough that an equivalent but "non-treated" control group at another institution could not reasonably be constructed. Hence, LEAD researchers used interviews and surveys of current and former SaS participants to determine what the impact of the program on them had been.
The interviews of current and former SaS participants began with the pilot interviews conducted in July and continued with additional individual interviews conducted by phone in November and December of 1997. In all, 11 undergraduate participants, 9 graduate participants, and 5 people who participated both as undergraduates and graduates were interviewed. These interviews provided in-depth information about program participants' backgrounds, educational experiences, and aspirations; their experiences being recruited into, participating in, and in some cases, administering the SaS program; and their perceptions of the intellectual, academic, emotional, and practical impact of the SaS program on them, their current life trajectory, and their future plans.
In December of 1997, 38-question surveys were sent to all 52 former and current SaS program participants who could be contacted to check the statistical reliability of the information we had gathered through interviews and to collect additional information about how participants rated and evaluated certain aspects of the program. In all, 29 participants returned completed surveys by the time of our statistical analysis, for a response rate of 56%.
3. Outcomes from the Tracking Data for Undergraduates and Graduates
Overall, 68 students have participated in the Summer with a Scientist program since records were first kept in 1991, the vast majority of whom were majors in mathematics, computational science, and engineering. These participants have included minority undergraduates from around the country and female and minority graduate students from Rice University. Outcome information was available for 52 of these participants: 22 whose only participation had been as undergraduates, and 30 who had participated as graduate students. For the undergraduate participants, the main goal of the program is to encourage as many as possible to complete their baccalaureates and advance to graduate school. Figure 1 shows the outcomes for the 21 undergraduate participants who had already graduated and had the opportunity to make a choice about graduate school. As one can see, almost two-thirds of these participants enrolled in graduate school, while another third gained employment in a math- CS- or tech-related (MCST) field. By comparison, according to the Baccalaureate & Beyond Longitudinal Survey done by the National Center for Educational Statistics, less than 9% of African Americans and Hispanics who received baccalaureate degrees from 1992-1993 have gone on to enroll in graduate schools and, based on the data from other ethnicities, the percentage is presumed to be even lower for those with baccalaureate degrees in mathematics, computer science, and engineering.
For the graduate student participants, the main goal of the SaS program is to retain them in graduate school until they receive an advanced degree, preferably a Ph.D. Figure 1 shows the outcomes for the 30 graduate student participants who have participated in the program and have been tracked. The rate of retention so far is an impressive 97%, with just one student having left graduate school without a degree. A total of 17 (57%) of these participants are still in graduate school making steady progress toward their degrees, and of those 12 who have already graduated with advanced degrees, two-thirds have received Ph.D.s.
Clearly, SaS program participants are enrolled and retained in graduate school at much higher rates than is the case with the typical minority (or non-minority!) student in mathematics, computational science, or technology. The effect of the program itself on these impressive enrollment and retention rates will be explored in the section that follows.
4. Essential Elements for Success: How and Why the Summer with a Scientist Program Works
Although tracking the graduate school enrollment and advanced degree attainment of past and current participants in the SaS program allowed us to determine the extent to which the program's ultimate goal was being met, it is unclear from these numbers alone what the impact of the program itself was on participants' decisions to attend or remain in graduate school. Interviews and surveys of past and current participants were used to determine the perceived impact of the program on participants' readiness for, decisions about, feelings about, and success in pursuing graduate degrees. These two data sources were consistent in demonstrating that the Summer with a Scientist program had a significant and powerful influence on the choice to attend or remain in graduate school for the vast majority of students who participated in it. According to our survey, 89% of the 27 participants who returned surveys said that the program increased their desire to attend or remain in graduate school, with 15 saying that the program increased their desire "greatly," and 10 saying it increased "somewhat." Only 2 of the participants' surveyed felt the program had no impact on their preexisting feelings about whether or not to attend or remain in graduate school.
Similarly, many of the 17 program participants we interviewed said the SaS program had been a major factor in convincing them to enroll in or remain in graduate school. All four of the interviewees who had participated as undergraduates and had since graduated said the SaS program was the deciding factor in their decision to attend graduate school. Another undergraduate participant who had yet to graduate remarked, "Since I've been in this program, this is honestly the first time I've ever really been motivated to pursue graduate studies. I'm thinking about it. I might not do it, but it's definitely an interest that keeps growing." Several of the interviewees had never even thought of themselves as having the potential to enroll in graduate school until they participated in the program and were convinced by Dr. Tapia and their experiences in the program that they had what it takes to succeed as a graduate student. The program also had a powerful influence in convincing a number of graduate student interviewees to remain in graduate school when the going got tough and their motivation for continuing was in question.
Participants were also asked more specifically about which parts of the program influenced their decisions to attend or remain in graduate school. Table 1 shows how surveyed participants responded when asked to rate on a 5-point scale how the various program activities affected their desire to attend or remain in graduate school. The activities are listed in order from those with the greatest positive influence to those with the least influence. The three most influential activities in increasing participants' desire to attend or remain in graduate school were: interactions with the program director, being in the company of other minority students, and the interactions with the other students in the program. All nine of the activities listed caused some degree of increase in the desire to attend graduate school for at least half of the participants surveyed.
Table 1: Impact of various program activities on surveyed participants' desire to attend or remain in graduate school
The SaS program has other impacts beyond encouraging participants to attend or remain in graduate school. According to interviewees and survey respondents alike, participation in the SaS program created the feeling of belonging to a community, increased their feelings of preparedness for graduate school, and increased their feelings of preparedness for a career. Furthermore, the program increased most participants' confidence in everything from doing research, giving presentations, discussing controversial topics in a constructive way, and mentoring other students to handling graduate levels work, completing their degree, and succeeding in their careers. Decreases in confidence due to some activities (most notably, doing research) were seen for only 3 of the 27 participants surveyed.
In the sections that follow, evidence from both the interviews and the surveys will be brought to bear on the question of how the SaS program produced these desirable effects. By performing an integrated analysis of the qualitative and quantitative data, we were able to derive hypotheses about the essential elements of each of the three critical structures in the Summer with a Scientist program: the program director that administers and sustains the program, the community of fellow minorities that surrounds students in the program, and the research project that is the focus of the participants' summer training. The complete evaluation report includes citations from the graduate student retention literature consistent with these hypotheses.
4.1. Essential elements of the program director The role of the program director in the success of the SaS program is an important one. As seen in Table 1, 89% of survey respondents said their interactions with the program director increased their desire to attend or remain in graduate school-59% "greatly" and 30% "somewhat." Because it is the program director's and NPACI's desire to see this program replicated elsewhere-and with directors other than Dr. Tapia at the helm-it is necessary to determine which of the program director's personal, situational, and background characteristics and which of his program-related activities have contributed to the remarkable success of the program at Rice. Once these characteristics and behaviors are explicitly identified, they can be sought and encouraged in those who wish to direct similarly successful programs at other institutions.
Our analyses determined that there are six characteristics of the SaS program director that are essential to the demonstrated success of the SaS program at Rice University. In brief, the program director helps to make this program a success because he:
Student interviewees discussed how important it was to have a role model showing them it was possible for minority students like them to achieve their academic and career goals. They needed to see an academic who had encountered struggles similar to their own-someone for whom climbing the academic ladder had not come easily-in order to believe that they, too, could succeed. It was also important that this person be approachable so that the minority students felt comfortable and confident enough to go to him for support and advice. The SaS program provided participants with an ideal faculty role model in the person of its director, Dr. Tapia, and as a result, many came to feel less alienated and less anomalous in the predominantly-white academic culture they faced at Rice University and elsewhere. The program director's role as a mentor is also important. All of the interviewees discussed how he supports and encourages students through individual mentoring relationships and builds their confidence to pursue and persist in graduate degrees by believing in them even when others-including themselves-do not. His belief in the students' abilities is essential for their development of confidence either in their pursuit of a career in the sciences or in their persistence in graduate school.
Another essential element is that the SaS program director serve as a guide to the academic world by providing professional development information and opportunities to participants. In both his individual mentoring relationships and in the SaS community, the program director instructed students on what to expect and how to behave in a professional and academic environment. Equally important was that he has influence and "clout" within the university so that he can actively support and advocate for the students' academic and financial interests. Finally, the program director's active involvement in creating and sustaining the student community was integral to its function as a permanent and influential support system for minority graduate students at Rice.4.2. Essential elements of the SaS student community
The role of the SaS student community in the success of the program cannot be underestimated. While the program director plays the role of mentor and guide for many of the students in the program, the size and cohesiveness of the SaS student community allows support to come from multiple individuals rather than a single mentoring relationship. As a result, students within the community can "fill in" for the program director in circumstances where he is not available and can gain their own valuable experiences as mentors. As seen in Table 1, the presence of, and interactions with, other minority students in the program greatly increased the desire to attend or remain in graduate school for 41% of those surveyed and somewhat increased the desire of another 48%. This is consistent with research on graduate school success and retention, which has found that interactions with peers (as well as interactions with faculty) were a more important determinant of success in graduate school than were undergraduate background and personal characteristics (Blackwell, 1987; Girves and Wemmerus, 1986; Hartnett, 1976; Nettles, 1990). Minority students often feel outside the mainstream of their departments and often have problems interacting with faculty members (Carrington & Sedlacek, 1976; Allen et al, 1984; Clewell, 1987; Nettles, 1990; Smith & Davidson, 1992; and Turner & Thompson, 1993). Therefore, for minority students in particular, interactions with graduate student peers may provide an otherwise difficult-to-obtain connection to their department and to other researchers. Within the CAAM department at Rice University, the SaS student community provides a vibrant and ongoing forum for such interactions to occur.
Our analyses of survey and interview data determined that there are eight interrelated characteristics of the SaS student community that are essential to the demonstrated success of the SaS program. This multi-ethnic student community aids in minority student recruitment and retention because it:
Because both graduate and undergraduate students participate and because participants are involved in the mentoring of minority high school students, the SaS community provides a matrix of mentoring relationships between students at different levels so that each student can both be a mentor and be mentored. Hence, not only are role models and guides provided by the students and professors that are "ahead" of a student, but each student becomes a role model and guide to those "behind." Many interviewees emphasized the importance of feeling that they were giving back to their ethnic communities, a "duty" that the program director himself repeatedly emphasizes. Minority students often feel this need to give back to their community in order to demonstrate that they have not abandoned it. The SaS program gives multiple opportunities for such reciprocation to occur, regardless of the student's level or length of time in the program. Participants in the SaS program expressed the belief that both the mentor and the mentee are retained in such a relationship. Such a community also allows support to come from many individuals rather than having students rely on a single mentoring relationship. The program surrounds students with successful graduate student role models and with caring peers who are willing to provide guidance and advice.
Another essential element of the SaS program is the fact that most students involved in it participate for multiple years. This allows a continuity and sense of permanence that few other summer programs have. Our analyses of participant interviews made it clear that the very reason that the SaS community has become so strong and influential is that it is sustained through having students participate for multiple years. As a result, the social infrastructure that is in place is permanent and self-perpetuating, and the community and its culture can be indefinitely maintained. SaS repeaters are the bearers the community; it need not be created anew every summer. In addition, the graduate student participants keep the community going informally during the rest of the school year. Furthermore, this community is fully integrated into the life of the department and receives legitimacy through its connections to a respected research center. This, too, adds to its permanence and influence.
In addition to providing participants with a sense of belonging and a multi-ethnic "home" within the university, the SaS community provides participants with an open forum for discussions about race and ethnicity. These discussions, along with instruction, practice and encouragement in professional development occur during the afternoon meetings held weekly. Interviewees expressed that these meetings were essential in teaching them how to have discussions of controversial topics in a constructive way, how to tactfully disagree with academic colleagues, how to explain and give presentations about their work, and how to conduct themselves as professionals in an academic setting. As seen in Table 1, the discussions about professional development increased graduate school interest and retention for 79% of the SaS participants surveyed, and the discussions about race and ethnicity increased graduate school interest and retention for 74%.
4.3. Essential elements of the research project
Since participants in the SaS program spend the majority of their time working on their summer research project, the project is, for most, the focal point of the program. For the undergraduate interviewees in particular, the research component of the program was key to the program's impact on them and their decisions regarding graduate school. Furthermore, as seen in Table 1, the experience of doing research in a university setting played an important role in encouraging nearly three-quarters of the SaS survey respondents to want to enroll or remain in graduate school.
Our interview and survey analyses determined that there are five characteristics of the summer research project that significantly contribute to the demonstrated success of the SaS program in graduate student recruitment and retention. In brief, the research project is integral to the Summer with a Scientist Program because it:
A number of SaS interviewees also expressed that their research projects increased their knowledge of their discipline and gave them a more concrete understanding of how the things they had learned in classes could be applied to real world problems. For students who thought careers in mathematics meant careers as teachers or theoreticians, working on an applied math project in the SaS program was a happy revelation that reinvigorated their interest in their discipline. Also, once SaS participants had a greater understanding of their discipline, they were better able to make choices about their careers and what, if anything, they wanted to study in more depth. In a number of cases, projects began during a summer in the SaS program developed into a graduate student's dissertation research.
The new understanding about research and life as a graduate student that undergraduates in the program gained made it easier for them to make choices about enrolling in graduate school themselves. Participants had only to observe and ask questions of those around them to get an accurate picture of the life of an academician and researcher and to get a clear view of the path that lie ahead if they chose to enroll in graduate school. Finally, working on a research project with a faculty mentor provided participants with a forum for collegial interaction with a faculty member. This gave them the confidence to approach professors and discuss research topics even after the summer had ended.
The evidence in this document establishes the success of the Summer with a Scientist program with respect to the recruitment of minority undergraduates into graduate school and the retention of minority graduate students at Rice University. Tracking of student academic outcomes, and interviews and surveys with student participants demonstrated not only that SaS participants are enrolling in graduate school and obtaining graduate degrees at an unusually high rate, but that most of these participants feel the program had a powerful impact on their decisions about and success in pursuing advanced degrees. A number of them asserted that they would not have completed their degrees-or thought to enroll in graduate school at all-had it not been for their participation in the Summer with a Scientist program.
In our analyses of survey and interview data, we have sought to identify the essential elements that were critical in bringing about the remarkable success of the SaS program. Because of the interest in replicating this program at other sites, we attempted to determine what characteristics of the program director, the student community, and the research project would likely have to be present in order to bring about similar outcomes in other institutional and departmental contexts. In short, we have attempted to provide guidelines for how to replicate this program at a department that is not associated with the CRPC, a University that is not Rice, and a program director that is not Dr. Tapia. The three lists of essential elements should be seen as hypotheses about what is needed for a similar program to succeed in another context. These hypotheses remain to be tested through replication.
Those wishing to replicate the success of the SaS program should keep in mind that the essential elements of the SaS program delineated here do not work in isolation but rather interact with each other to produce the desired outcomes. Also, this list of elements should be seen as a guideline for what to aim for in constructing a similar program in another departmental or institutional context, not as an unalterable blueprint. Failing to incorporate one or more of these elements may weaken the new program's chances for success, but not necessarily, and contexts very different from those at Rice may require that some elements not included on this list be added in order to achieve success. Until similar programs incorporating most or all of these essential elements are instituted in other departments or at other universities, our knowledge of the adaptability of the SaS program and the weight to be given to each of its "essential" elements remains incomplete.
Finally, although the results of this evaluation have shown the Summer with a Scientist program to be remarkably successful at achieving its stated objectives, readers should note that such favorable outcomes have taken many years to achieve. It is rare for a program to show successful outcomes in its first year or two, especially when the program is one that is being adapted for use in a new context. The process of determining the unique benefits and constraints of a new context and adapting to those constraints takes time. Those wishing to achieve success as quickly as possible are advised to incorporate formative evaluation and tracking of program participants into their program from the very start. The program administrators must then be willing to use this formative feedback to make whatever refinements and adjustments are necessary. In the end, those programs whose administrators are both patient and flexible will achieve the greatest success.
Sites & Affiliations | Leadership | Research & Applications | Major Accomplishments | FAQ | Search | Knowledge & Technology Transfer | Calendar of Events | Education & Outreach | Media Resources | Technical Reports & Publications | Parallel Computing Research Quarterly Newsletter | News Archives | Contact Information