Running Head: Senior Transitionprogram Evaluation 1

Running head: SENIOR TRANSITIONPROGRAM EVALUATION
Evaluation of Senior Transition Program at DePaul University Loyola University Qualitative Approach Alicia A DeLaRosa and Reina M Salcedo
SENIOR TRANSITIONPROGRAM EVALUATION
Table of Contents Program and Evaluation Overview 3 Introduction 3 Theoretical Framework ..3 History.4 Program Objectives 7 Stakeholders ...8 Program Content 10 Evaluation Logic Model .11 Assumptions and External Factors .12 Evaluation Purposes ...13 Evaluation Description ...14 Quantitative Approach 16 Quantitative Narrative .16 Survey Description ..17 Survey Administration .18 Statistical Analysis ...19 Data Presentation ..20 Qualitative Approach 20 Qualitative Narrative .20 Participants 22 Instrument ..23 Implementation ..23 Analytical Approach ..25 Presentation of Qualitative Results 25 Limitations .26 Budget 26 Timeline .27 Next Steps ..28 References ..29 Appendices .31 Appendix A: Logic Model ..31 Appendix B: Pre-Program Survey Email ....32 Appendix C: Pre-Program Consent Form &Survey 33 Appendix D: Pre-Program Survey Construct Map ..42 Appendix E: Post-Program Survey Email ...43 Appendix F: Post-Program Consent Form & Survey ..44 Appendix G: Post-Program Survey Construct Map 55 Appendix H: Focus Group Invitation Email 56 Appendix I: Consent Form ...57 Appendix J: Focus Group Protocol ...59 Appendix K: A Priori Construct Map ...62 Appendix L: Budget ..63 Appendix M: Timeline ..64 Appendix N: Poster Presentation ..65
Program and Evaluation Overview Introduction Founded in 1898, DePaul University is the largest Catholic university in the nation and the largest private institution in Chicago, serving over 25000 students (DePaul, 2013). With two campus locations, one in Lincoln Park and another in the Loop the Office of Multicultural Student Success (OMSS) is called to work with low income college students, first generation college students and students of color by developing sustained development programs. The office manages retention, persistence and empowering programs while also serving as an advocacy office for marginalized college students. One program that OMSS has piloted this academic year is the Senior Transitions Program. The purpose of this evaluation is to determine the success of the Senior Transitions Program. The definition of success is this context will be explained later in the narrative. Theoretical Framework The Office of Multicultural Student Success at DePaul University seeks to cultivate student success for students of color, low income college students, or first generation college students through sustained developmental programming, financial resources and advocacy. The department however has witnessed an increased need to assist students in their post college lives. Students are requesting assistance to best prepare themselves for their lives after DePaul. The office of Multicultural Student Success has aimed to create a program that will better equip its graduating seniors for post college life. The Office of Multicultural Student Success has initiated a senior transition program that will target graduating seniors involved in post college success programs to help alleviate some of the problems students are facing; such as preparedness for
graduate school or the workforce, what it means to be a graduate in society, and finding meaning in their own lives. The Senior Transition Program was put in place as a way to provide a series of capstone experiences that will prepare seniors as they approach graduation. The creation of a senior transition program was necessary because very little is known about what under-represented students do after graduation (Goodwin, 2006). Since so little is known, the program is intended to meet the needs of this diverse at-risk group of students; which is challenging. This program is one of the last opportunities staff have to prepare students for post-college life; a critical component to student success. Studies have shown that under-represented students more so dealt with isolation and a loss of identity when transitioning into their post-college life (Fox, 2011). Based off of this research, learning domains need to be addressed to assist in the post college transition. Creating a Senior Transitions Program that meets a wide variety of needs of this diverse group of students is challenging. This program is one of the last opportunities staff has to prepare students for post-college life; a critical component to student success. The Senior Transitions Program is an opportunity for graduating seniors to gain a better understanding on life after DePaul, both professionally and personally. The overall success of the program depends both on the OMSS staff as well as the students participating in the program. History of the Need It is important for colleges and universities to increase their services to prepare seniors to smoothly transition them from college students to emerging adults (Fox, 2011). Institutions have a responsibility to provide students with the necessary resources and experiences that will help students deal with this new time period in their lives. This is done through a variety of
target audience specified programs. One large audience and focus in current research are seniors. The senior year experiences are very critical will trying to evaluate how successful your students are after they live a particular institution. Are they prepared to adapt to the workforce or seek a higher degree? Do the students feel that they know what is necessary to attain a job? These are all questions that are on the minds of many Higher Education staff and faculty, policy makers, and students. At the end of their college experience seniors embark on a crossroad that challenges them to deal with a sudden shift in their identity. It is during this time that students face a time of feeling unprepared, a time for a new kind of learning and a time of tension (Olson, 2010). This can be a high anxiety time period for students. Because of this, there is a need to ensure that students are prepared and learn about a wide range of life topics including finances, living situations and wellness (Walls, 2002). This is a serious concern because many college graduates lack essential skills to succeed in todays global economy. The goal of institutions is not to simply graduate students but to have them be successful after college as well. Senior year is when students need to reflect on their new transition from college to student to emerging adult. It is during this time that seniors should begin an active investment in developing their identity and understanding new meanings for their careers (Stokes & Wyn, 2007). Seniors will need help in adapting to the changes that they are about to encounter. They will encounter a loss of expectations, deteriorating social relationships and a lost sense of identity (Fox, 2011). All of these combined can make the transition to post college life a very difficult one. In 2009 a College Senior Survey was conducted by the UCLA Higher Education Research Institute showing that four out of five seniors indicated that enhancement knowledge
was very important to them (Franke, Ruiz, Sharkness, DeAngelo, & Pryor, 2010). The study showed most graduating seniors placed high importance on the expression of personal values, creativity, initiative and leadership potential (Franke, et al., 2010). This is important because it shows that students are interested in their own personal growth and development. Therefore, it is important for colleges and universities to harness this particular stage of student development and discernment to ensure students are successful after college. In order to help college seniors there is a need to provide assistance for their transition prior to graduation. Prior research has shown that to achieve a successful transition, students need to renegotiate their networks and support groups (Arnett, 2000). Students often have to break ties with their current networks to make connections that can help them transition to adulthood. By connecting with other peers and early career alumni they create a sense of belonging through shared experiences, and as a result students feel more stable and secure (Fox, 2011). This is a serious concern as students are encountering new challenges and frustrations, and, at the same time, trying to understand who they will become as individuals, as professionals, and as employees (Olson, 2010). What has been found as a result is that there is a lack of senior transition programs made available by institutions to students. College freshmen enter college feeling the same way yet they are provided with programs and resources to help them transition, however, seniors do not get the same transition services (Hartel, Schwartz, Blude, & Gardner, 1994). Literature suggests that colleges and universities have to go beyond the traditional celebratory programs and events for seniors. Employers are pushing institutions to create experiences to help college seniors (Fox, 2011). This can be accomplished by creating a curriculum to create senior retreats, workshops, and alumni connections to address the issues of unexpectedness, sudden loss, and finding
networks of support that will help seniors with their college student to emerging adulthood transition. Program Goals & Learning Objectives The senior transition program aims to create a series of senior capstone experiences designed to help participants transition from college to post-college to emerging adulthood leading to successful discernment and preparation for life after college. Through workshops, field trips, and one-on-one career coaching, the OMSS staff, campus and community partners will provide students with the necessary skills and tools in order to be successful after graduation. The main goal of the program is to successfully prepare graduating seniors for their departure from DePaul. The program will strive to do three things. First, it will assist graduating seniors to learn how to successfully transition from college to post college life. Second, it ill challenge and support inactive seniors to take action toward career attainment. Finally it will engage graduation seniors in the process of discernment. The hope is that all students who engage in Post-College Success programs will graduate from DePaul prepared for career success, graduate school admission or post college service work. This program was initiated to improve the success rate of these at-risk students. In this case, success is accomplished by one of two things. First, students in this program will develop a career plan by exploring and engaging in various career-related opportunities and programs that will enhance their self-awareness and career choice understanding. Second, students who undergo this series of programs and workshops will be employed in a new job, enrolled in graduate school or post college service within a year of graduation. These two components are what determine student success. The office has developed a number of program goals and
learning outcomes aimed to achieve student success.
The first departmental outcome is that students who engage in OMSS programs and services will design personal financial plans. As a result, students will be able make personal financial management decisions and demonstrate financial literacy. The second departmental outcome is that students who engage in OMSS programs will be able to construct career plans towards post college career success. Participants will be able to choose career objectives, identify strategies for career mobility and success and comparing and contrasting post college planning. The final departmental outcome is that students who engage in OMSS programs and services will be able to apply concepts of social responsibility and leadership. Students will learn to integrate their personal values to social responsibility and leadership. There are several program goals that the office has put in place as well. These goals are broken down into three categories: routine, strategic, and innovative. Routine goals include identifying and recruiting graduation seniors to participate in the Senior Transitions program, developing and implementing a senior retreat and providing career coaching to students. Strategic goals include collaborating with Alumni Relations, creating a database to track student participation and develop a fast track senior curriculum for students who have been labeled as inactive. Future innovative goals that have been considered are creating a senior capstone course and creating revenue to support post college success initiatives. Stakeholders Stakeholders for this evaluation project are based heavily on the targeted student audience. The particular group being targeted is graduating seniors whom are involved in other post college success programs offered by the Office of Multicultural Student Success. Therefore, the primary stakeholders will be the Office of Student Success staff. The staff is comprised of an assistant director, a MOC post-college success guide, a graduate assistant, and a
STEP college success guide. The Assistant Director will be responsible for the oversight of the entire post college success curricular team. Each team member will focus on specific initiatives for example; the coordinator will oversee the senior soiree and career coaching. The graduate assistant will assist the coordinator and also oversee the senior retreat. Additional stakeholders include the senior students, alumni, campus partners, and community partners. Campus partners will include the Career Center, Alumni Relations, and Financial aid office. These three campus offices play a large role in preparing students for their transition. Workshops will be held by each office to address topics such as student loans, networking and resume building. There will be a large effort to reach out to outside partners such as businesses and companies so that students have the opportunity to visit sites and learn more about organizations. Stakeholders are various individuals and groups that have a direct interest in and may be affected by the program being evaluated or the evaluators results (Fitzpatrick, 2003). In this case the direct stakeholders are the career center, alumni relations, and financial aid office. We are aware that there are higher administrators and groups that will also be interested in learning about the project such as university policy makers, advisory boards; however our main stakeholders that we listed earlier is who we will be working with directly. The intended beneficiaries are the students; however, there is also much to gain for the office, which is ultimately aiming to create well-prepared citizens and active alumni. Finally, the program also affects the larger DePaul community. When students graduate and are successful, the DePaul community gains new alumni and connections. All of these groups are important because they hold a stake in the future direction of the program (Fitzpatrick, 2003). These stakeholders are important participants because the OMSS
10
staff will determine the future of this new program. The staff will decide what necessary changes if any need to be made to the senior transition program. These individuals are the determinants of financial revenue for this program and the services it provides to students. In addition, they will decide what changes if any need to be make to better assist graduating seniors; as well as what focus areas to zone into. Lastly, these individuals will determine whether any additional factors need to be considered. Program Content The Senior Transition Program is in its first year of existence. During this pilot year, assessment is being done to determine the needs of this particular student population. The career center is a critical stakeholder because the mission of the office is to ensure that students have career capital by the time that they graduate from DePaul University. Therefore, the senior transition program will focus heavily on career capital. Throughout this first year, the program includes one on one career coaching with the professional staff from the Office of Multicultural Student Success, career workshops, and trips to various businesses and companies. Business and companies will be selected based on alumni connections; that have been made via the alumni relations office. The program will be developed around the following content areas: alumni engagement and participation, senior recognition events, career action planning and attainment, and college to post college transition development. The participants will be given the chance to in a variety of experiences through the year. Activities will include career coaching, alumni connections workshop series, senior retreat, and senior soiree. Career coaching will provide participants the opportunity to engage in discussions about transition goals, reflections, and career action planning. Alumni connections workshops will each focus on different skills and a support group
11
that better prepare students for the transition. The senior retreat will include high impact activities and will be held off campus it is a great opportunity for students to build connections with other students. The senior soiree is an annual recognition event in which seniors and their families will be able to celebrate the end of the academic year. The staff has created several creative ways to incorporate these major components into the program and activities. For example in preparing students to design personal financial plans they will have to demonstrate effective decisions making skills and financial literacy. Students will do this by identifying common pitfall in money management and by reflecting on their financial needs versus wants. They have chosen to not merely provide information to these graduating seniors but really find ways to help them engage and practice their skills. Evaluation Logic Model The logic model (appendix A) is comprised of the different resources or inputs that are necessary for this evaluation plan to take place. Campus partners include other offices such as Alumni Relations, Career Center, and Academic Affairs, whom all have an impact on the success or failure of this program, along with the potential capstone course. Even though majority of the resources and time will be coming from the Office of Multicultural Student Success, these campus partners are necessary to ensure that the students are being successful throughout the academic year. Along with campus partners, there are community partners that will assist with the different career field trips that students will be able to attend to companies that would fall within their desired careers (i.e. Northern Trust Bank finance and business). The activities on the logic model are from the Senior Transitions Program and potentially in the Spring 2014 will also form part of the senior capstone course. Through these different activities, students will have the opportunity to explore more in depth their career choices and
12
whether they are gaining career capital through internships and networking. Some of the workshops include financial responsibility, resume and cover letters, and explaining the importance of social media in regards to their professional life (LinkedIn would be the primary source). These workshops will be ran either by the staff of the Office of Multicultural Student Success and from the Career Center to ensure that the information students are getting is correct. With every program or curriculum, there are always tangible outcomes that one desires to meet. The outcomes on this logic model vary from having students secure internships their senior year to building strong relationships with them so that as alumni they come back to ensure the success of the program. When looking at these outcomes, it is important to note that these outcomes are currently in place for the program but will also be important for the overall DePaul Campus community. Assumptions and External Factors When thinking of how the program will not only take place throughout the academic year but also how the different participants from students to all the stakeholders will react to the program, assumptions on level of participation are made. At the beginning of the program, participation from all participants might be higher and the assumption is that it will remain at a high rate. These assumptions do not take into consideration the responsibilities that everyone has and believe that everyone involved will be able to dedicate enough time for the program to be successful. In order for the program to be successful, or fail, there are external factors that affect the program beyond the control of those directly involved with it. Budget is a critical factor to whether or not the program continues. Without budget available for the program, workshops,
materials, and field trips would not be possible. This budget is directly tied to retention rates and
13
the number of students enrolled at DePaul. If retention decreases and students are not present, the program suffers due to a lack of numbers in students. It is known that every person is at a different developmental stage. This is no different in regards to career capital and current senior students. Students whom have been actively participating through their college career will more likely have a higher sense of what it means to have career capital along with actually having some career capital compared to their counterparts whom might now have participated as much or at all. Evaluation Purpose We have chosen to evaluate this particular program because this is the pilot year of the program. Since it will be the first year that the program is implemented there are a number of things we are hoping to discover. First, we want to assess the effectiveness of the Senior Transitions Program. Second, we want to assess the depth that the program prepares seniors as they move into their post college life successfully. Finally, we want to ensure that the learning outcomes and program goals are feasible and attainable. The major purpose of this evaluation is to assess the outcomes of this program to ensure that DePaul students are prepared to enter their lives after graduation. Therefore, we have decided to conduct an outcomes based assessment that utilizes a combined approach of formative and summative analysis. According to the Office of Multicultural Student Success there are three major challenges that the staff often faces when preparing students. The first issue is learning whether or not the workshops and program content are effective. The problem has been recognized but finding the appropriate way to transfer this new knowledge and information can be trickier. The staff would like to know if the program structure they have developed is being implemented as effectively as possible. The second issue is addressing how much change is seen in these students as they
14
progress through the program. Are their levels of comprehension improving? Finally, the staff would like to know if the learning outcomes and goals are feasible and attainable. Post college research shows that the majority of students are unprepared for the career force let alone for their lives after they depart from their undergraduate institutions. Through the pilot year of the Senior Transitions Program, there will be better results as to the success of the program with the students who are participating in the program. The Office of Multicultural Student Success will be meeting with students one on one, along with distributing surveys in order to find out what the students needs are in regard to their desired careers. The success of this program is captured in the program goals and objectives that were discussed earlier. Additionally, the success of this program is also heavily based on where students are a year after they participate in this program. Students who participate in PostCollege Success Initiatives such as the Senior Transitions Program will develop a career plan through the exploration and engagement in various career-related opportunities and programs that encourage self-awareness and career discernment. Through the implementation of their career plans, participants will be employed in a new job or be enrolled in graduate school or post-college service opportunity within a year of graduation. Evaluation Description The evaluation as I mentioned will be both formative and summative with a combination approach of process and outcome. This evaluation will be formative because it intends to improve the way that the program is delivered (Wholey et al., 2010). The stakeholders know that looking at the implementation of the program is important. The evaluation will examine what occurs during the program as well any results that may occur because of it. This particular area of assessment is to help gain a better understanding of whether or not the program is
15
attaining its intended outcomes. Using a formative evaluation will help not only improve the way the program is delivered but it will help facilitators recognize problem areas for these students (Upcraft, &Schuh, 2009). Combining a formative and summative evaluation was done purposely. While the formative aspect would help provide information that could influence decisions to make structural changes to the senior transition program; the summative aspect was intended to measure the program outcomes and impact. We hoped to gain a better idea of the ongoing impact that the program is having on students. A summative evaluation allows the tracking of the success rate of students. For example, have the students learned the appropriate information and is that as a result of their development throughout the senior transitions program. There are strengths and weaknesses to our plan. Because we will be focusing on process and outcome, we will not be able to focus on every single detail of the program. In addition, the needs of our stakeholders will be somewhat neglected because we are not utilizing a needs assessment. However, we hope the strengths of our combined approach will help make up for this weakness and provide some recommendations. Another strength, is the relation that the evaluators have to the program. Both evaluators indentify with the target population that this program works with and have worked with set target population. One of us is a graduate student who is currently working in the Office of Multicultural Student Success and the other is currently interning at a different institution working with the same student population. As a result, we are able to draw on our experiences as we design this evaluation. As I mentioned it is strength of our evaluation that we both identify as low income college students, first generation college students and students of color. It is important, though, that we recognize that this can also be a weakness. Given our perspectives, we naturally already
16
have preconceived opinions, so it will be difficult to claim complete objectivity. We address this here so that we can be held accountable to best minimize our biases. The specific processes and outcomes we will be investigating include the success of the program as students engage in a series of capstone experiences and complete their last year at DePaul. As previously described, there are specific ways that we determine and define success. If these areas of success drive the design of this session, it will be important to determine if it is being implemented as intended. Additionally, because the outcome approach is concerned with describing, exploring, or determining changes that occur in program recipients (Fitzpatrick, Sanders, & Worthen, 2003, p. 21), we will investigate how students have changed as a result of this session of orientation. The findings of how the program works and the experiences students are having will allow stakeholders to determine how to best support this particular student population transition to their post-college life. Quantitative Approach Quantitative Description The quantitative portion of this evaluation will consist of surveying senior students who are currently involved in the senior transitions program. We will use census sampling, which is a sampling technique in which the entire population will be surveyed (Wholey et al., 2010). In
our case, the entire population consists of 87 seniors. We are aware that not all the current seniors will respond to the outreach of completing the surveys and from past evaluations that the office has conducted we have anticipated that there will be a 17-20% response rate from the 87 graduating seniors, which would be from 15-18 students; however, since our sample is small compared to that of the office and university, we hope to have an increase in our response rate.
17
Two surveys will be administered one at the start of students senior year and one immediately after the final senior capstone experience of the senior transitions program and require it to be completed within one week. The pretest will be given to the students at the beginning of the Fall quarter (September). The focus of the pretest is to find out what the students hope to gain from participating in the Senior Transitions Program throughout the year along with the relationship, if any, that they have had with the OMSS staff and faculty. The posttest will be scheduled at the end of the academic year sometime in late April or early May. The posttest will focus on what students have gained from the program and the relationships with the OMSS staff throughout the year. In addition, the post survey will allow students the
opportunity to volunteer to participate in a focus group. Our research design is quasi-experimental because students will not be chosen randomly to a treatment or control group. Having a control group does not make sense for this evaluation. The evaluation and assessment will take place during the second year of the program; therefore we are more concerned with the process and outcomes as opposed to a control group. However, we hope to have a control group and even have a random sampling to identify additional factors that may impact a students transition to post-college life. The evaluation will also be a
longitudinal because we will be surveying students for a particular length of time. Survey Description The pretest for the students consists of 15 questions (Appendix C), which range from groupings of post DePaul plans to comfort with cohort; these grouping are further broken down to understand the plans students have in regards to job searching or graduate school. These questions shift between Likert scale (strongly agree to strongly disagree; 1-5) and fill-in questions to get a better understanding of their responses. As students progress through the
18
survey, the questions will start developing into relationship building and reflection questions, which are aimed at getting students to critically think of their experiences in the program. In addition to these, the fill in questions ask for clarification on their plans and if they have applied for jobs, graduate programs, or plan on taking some time off. The posttest has a very similar format to that of the pretest and also contains 15 questions (Appendix F). However, these questions are aimed to understanding how the program has benefited the students and if additional resources would be necessary to be prepared for life after DePaul. At the beginning of all each survey, individuals will be asked to electronically sign that they acknowledge that the information gathered is to be used for research by the OMSS staff in order to assess the Senior Transitions Program and whether a senior capstone course is necessary for students. At the end of the post-test there will be an option for participants to volunteer for one of two focus group sessions. All individuals will be asked for their university identification number along with their email address. These questions will be asked at the beginning of the survey; however, all demographic questions will be at the end of the survey so that this does not affect the responses given. Since these are both short surveys, they will be web-based and should last between 15-20 minutes to complete. Survey Administration As previously mentioned, all both surveys will be web-based and will be administered via email. The pretest will be administered in September and the posttest will be administered at towards the end of the Spring quarter (April or May). The main OMSS office email will be utilized to administer the survey. The results will be automatically collected via the web-based tool. The student workers will be responsible for collecting the surveys and tracking the amount completed. Once the deadline has been met for each survey, student workers will notify the staff
19
member that is in charge as well as the evaluators. Student workers will also be responsible for sending the email reminders for individuals to complete the surveys along will distribute the incentives, $5 gift cards to Starbucks. There will be a total of 20 gift cards distributed to individuals, 10 for the pretest and 10 for the posttest. Through the administration process of the surveys, there will be pilot testing of each survey that will be administered to all of the OMSS staff, including undergraduate interns. This pilot test is to ensure that all the links to the surveys are working properly and to give the entire OMSS staff an opportunity to interpret the questions and ask clarification questions before going live to all participating individuals. The reasoning for the undergraduate interns is to get the student perspective of the interpretation of the survey. Statistical Analysis Since the population of the current students is already limited to those participating in the Senior Transitions Program, we will be using factor analysis, multi-regression, and crosstabs. Through these different analyses, we hope to gain a better understanding of how students are processing all their experiences within the program along whether the outcomes of the programs have been met. However, since the target population of OMSS is low-income, first-generation, and students of color, there is a higher possibility that majority of the responses will be students of color. As part of the analysis of the responses, we will be looking at what other co-curricular or academic programs individuals were involved in to determine whether this might have had an effect on their responses. As stated previously, the graduating seniors will be examined over time.
20
Data Presentation All of the data obtained through all three surveys will be entered through SPSS. Since most of the responses are Likert scale, we will be using cross-tabulation for those. The fill-in responses will be categorized and assigned a number through SPSS in order to be able to compare responses. The charts will obtain the responses that individuals gave in the surveys along with a visual representation of the demographical identities of these individuals will be visualized through pie charts. The fill-in responses will be compiled into bar graphs to see how the responses across all three surveys differ. Tables will also be used for the responses that were given through the Likert scale. These responses will have been displayed through the tables and charts. The tables serve to put the raw number of responses available whereas the charts will function to better understand how different factors (i.e. co-curricular activities, gender, major) affected the responses given. Qualitative Approach Qualitative Narrative In order to obtain a more in-depth understanding of the results of the survey, focus groups will be conducted one to two weeks after the post survey results have been collected. Facilitating focus groups will help collect data that is not captured within the survey and that students may not have felt comfortable sharing. Qualitative data allows evaluators to gain a deeper insight into how programs work or fail to work (Wholey et al., 2010). However, the larger benefit is that qualitative data techniques, such as focus groups provide the opportunity to pay closer attention to the way students describe their experiences within the program instead of merely collecting statistical information.
21
Qualitative data therefore is imperative as a supplemental tool to our quantitative data as it can present a clearer picture of the members and statistical information that the survey provides (Wholey et al., 2010). A qualitative approach will allow us to gain a better understating of the success of the program specifically whether or not program participants felt that they gained career capital for life after DePaul. The career capital that we hope to gauge involves evaluating self-efficacy, communication, and confidence building skills. In addition, we anticipate that we will have a chance to ask students for suggestions and areas of improvement. Our goal is to be able to base the success of the program on whether or not the outcomes were achieved so that modifications can be made for future implementation. For the purposes of our program assessment, we have chosen to form two focus groups. The groups will be formed to further conclude how students felt about the Senior Transition Program and its components throughout the year that cannot be fully answered by our Likertscale option. The focus groups will be discussing process and outcome related questions and will address students feelings and attitudes regarding the programming and resources that were provided. The program outcomes that were listed to determine program success will be addressed within the three sections of the focus groups. Impact related questions will be addressed to explore a range of impacts both unintended and unanticipated and what they meant to the students if any (Wholey et al., 2010). In addition, there are process related issues that will be addressed to discuss how they might be improved as well as to check compliance with best practice by using student reviews (Wholey et al., 2010). It is impossible to determine what will come from the focus groups. However some assumptions can be made. For example, it is likely that participants of the focus groups will have a wide range of attitudes and opinions towards the program that perhaps could not be fully
22
grasped on a survey. While we have drawn up a focus group protocol (appendix J) we are well aware that there may be issues or comments that are brought to the table that may require more time than what was allocated. The facilitator therefore will be well aware and prompted to follow the developed protocol. The protocol questions are made to relate back to the pre and post surveys that students were asked to take. The focus group protocol will hopefully provide an opportunity to gather more detailed responses and clarify any confusing issues. However, the facilitator will be made aware that there might also be instances in which his/her judgment will best determine the need for additional questions and time. Participants Focus groups are a great way to gather information. Focus groups are beneficial because the goal is not to reach a consensus or a solution, the environment is nonthreatening and participants are homogeneous (Wholey et al., 2010). Students will have the opportunity to volunteer to participate in the focus group. Thus it will be a convenience sampling, which will make it easier to recruit students as we will be working with those who are accessible the day of the focus group sessions (Upcraft, &Schuh, 2009). At the end of the post Senior Transition Program survey, we will incorporate an additional section where we will encourage and solicit volunteers for our focus groups. The section will state This was the first year of the Senior Transition Program; therefore we are working on improving our program for future students. We welcome your feedback and encourage you to participate in one of our focus group sessions. Please check the box below and provide your email address if you are interested in having an open discussion about your recent program experience and would like to enjoy free lunch on DePaul.
23
Students who indicate that they would like to participate in a focus group will be selected through a census survey. Web-based surveys have made it much easier to survey an entire population; which is a benefit for us since our pre and post-surveys are both web-based (Upcraft, &Schuh, 2009). Participants will not be selected until the survey is completed and analyzed. The number of focus groups will depend on the number of volunteers; however we do not anticipate any more than two groups. Each focus group will consist of eight to ten students and will last from forty-five to sixty minutes and will occur on two different dates at different times to give more students the opportunity to participate. Instrument At the beginning of the focus group, an introduction with some conversational questions will be asked to relax the participants; very similar to a shortened ice breaker. Following the introduction, the facilitator will explain the purpose of the focus group session and will address what the goal of the night is. The participants will then be asked to fill out and sign their consent forms (appendix I). After, participants have signed their forms, they will be asked to introduce themselves and state their major and future plans (if any). The facilitator will then begin the session using the guidance of the protocol tool. Questions asked will be descriptive, casual, value and action questions. Throughout the session, students will be encouraged to feel comfortable and encouraged to participate. We have chosen to pilot the focus group session to the OMSS staff to gain a better understanding of the clarity of questions and language used. The hope is that the pilot testing will allow us to make changed as seen necessary. In the future we hope to be able to pilot both the surveys and focus groups to students but as of right now that is not possible. Since, it is the
24
first year of the program we are more concerned with what is going on with the program as opposed to control group data collection. Implementation In hopes of getting the most accurate information from students, the focus group sessions will be conducted one to two weeks after the quantitative data has been analyzed. Sessions will therefore take place at the end of the school year right before graduation. The students who had previously shown interest in participating will be sent an email invitation for the focus group, which will contain a reminder of their interest. Since, we are aware that it will be close to graduation and things may come up, we will send a reminder three to four days prior and then again one day before the session. The focus group sessions will occur in the OMSS conference room at noon, but time is subject to change if student availability is low. Both sessions will occur during the same week but on different days. Lunch will be provided as an incentive for participation (Wholey et al., 2010). The facilitators for the session will be the two head evaluators, Reina and Alicia. Reina and Alicia are a great pick because there is less bias than if an OMSS staff member were to facilitate. Reina and Alicia will decide who will take what role. One facilitator will have to be the note-take and observer as well as keep track of the audio recording while the other facilitator will be asking the questions and moving the conversation along. The facilitators will be trained for both roles since it is understood that neither role is easy (Upcraft, &Schuh, 2009). The pilot session will serve as practice beforehand to ensure a smoother facilitation for the two focus group sessions that will take place with students (Wholey et al., 2010).
25
Analytical Approach After the focus group sessions have occurred, the evaluation team will collect all that data and begin to transcribe it. Software tools such as invivo will be used to help transcribe the focus group notes and audio recordings. Prior codes will be determined beforehand based on topics and themes (Upcraft, &Schuh, 2009). Themes include skills, learning experiences, and most valued aspect of the program, graduation and career choices. After the initial coding occurs, evaluators will once again go through the documents to see if there are any additional underlying topics and themes (Upcraft, &Schuh, 2009). Three methods will be used to ensure that the data is being interpreted appropriately. First, inter-rate reliability will be used to make sure that coders are consistent. The coders will code at the same pace and asked to take specified breaks to confer with other coders. This checking code process will allow the identification of any potential discrepancies and create congruency (Wholey et al., 2010). Second, member checks will also be utilized. The team will summarize through themes and release them to participants via email with encouragement to provide feedback about the accuracy of the results. The only problem with this method is that participants do not always provide the most accurate feedback (if they provide any feedback at all) (Wholey et al., 2010). Lastly, peer debriefing will also be utilized. An additional Student Affairs staff member that is not familiar with the study will be asked to review findings and interpretations. The hope is that this individual will not only provide an outside perspective, but this perspective will also be unbiased (Upcraft, &Schuh, 2009). Presentation of Qualitative Results A full report will be provided to our primary stakeholders. Additionally, a condensed, more focused report may be provided to any additional stakeholders as well as administrators
26
and staff, such as directors, deans, and advisors. Included in our final report to the OMSS staff will be an InfoGraph of main themes, outcomes, and examples of student responses. The InfoGraph will be an additional tool to display the relationship between student responses and their relation to outcomes stated in the logic model. All stakeholders will have access to the raw data; direct quotations, codes, common themes; topics and words in excel spreadsheet tables. Graphs with the count of specific topics, answers, and words will also be provided as a visual source. There will also be the option to request the text of audio recordings.
Limitations As we mentioned in earlier sections there are a number of limitations with our evaluation plan. The first limitation is the fact that we will be unable to conduct a breakdown of factors and identities to identify differences amongst the student population based on race or gender. A second limitation is the fact that we do not have a control group. Therefore, our results will by default be skewed as we have no comparison group. Another limitation is the fact that as it if the first year we are solely focusing on the process and outcomes of the program. There is much more in depth analysis that can be conducted but that is not the purpose of our evaluation project. Therefore intersecting identities and other comparisons such as race, and gender will not be the main focus of data analysis. It is important to keep in mind that the OMSS has had low participation rates in past evaluation and assessment efforts. The final limitation is the lack of student feedback and participation may be a limitation. Budget The goal is to utilize as little additional money as possible when performing this assessment. The materials used including paper, pens, and printing supplies will be from the
27
Office of Multicultural Student Success. Statistical analysis will also be made possible by DePaul because they will be providing (software). All other costs associated with the assessment will be incentives. For example, we will be purchasing twenty $5 gift cards to Starbucks to distribute randomly to those who complete the surveys. In addition, we will be purchasing lunch for focus group sessions. We estimate a cost of about $100 each time; we plan to order pizza and drinks. We estimate that there will be about 8-10 students per session so we hope to order no more than four x-large pizzas; enough to feed all the students. The focus group sessions will take place in the OMSS office conference room so there is no additional cost. We have budgeted for any additional printing materials we might need such as poster boards etc. In addition, we will be purchasing a recorder and foot pedal. As we the evaluators will be facilitating the focus groups that cost is included in our pay. Finally, student workers will perform the coding technique, utilizing the construct maps during their standard working hours within the office so no additional compensation is necessary. Timeline The timeline provided (appendix M) gives a more visual form of the entire assessment project from the planning stages to the implantation and next steps. The timeline is color coded by year, as this will take about a year and a half. The blue is for 2013, which are some of the planning stages. 2014 is coded in green and includes planning stages and some of the
implantation which will be done at the beginning of the school year (September 2014), this is the pre-test implementation. Finally, 2015 is coded with red which is the continuation of the implantation, specifically the post-test and focus groups along with the next step which is done during late summer (August and September).
28
Next Steps As seen through the presentation sides (appendix N) the next steps can be found on the last slide and are a few areas that will be useful for further assessment. The first step after this assessment project is complete will be to do a second implantation with the 2015 senior cohort with improvements based on the data collected from the project along with suggestions from all major stakeholders. Since the first implantation did not pilot the survey and focus group
questions to current DePaul students, this is a major part of the next steps. The data collected from the previous year is only looking at the outcomes and process of the Senior Transitions Program, but outside factors, such as race, gender, or first generation are not taken into consideration. Since we know that OMSS targets first generation, low-income, and students of color, this is something that is to be taken into consideration through a sample t-test and match comparison on various demographical factors. Knowing that there are many more students within DePaul then just those participating in this program, one of the final steps would be to have a control group in place, specifically students whom are not part of the program complete the surveys and attend focus groups to get a better understanding of the needs of students at a university level. By doing all of these steps, the last step would be to create a five year strategic assessment plan for the program in order for it to be successful. This five year strategic assessment plan will be done with the OMSS staff and major stakeholders.
29
References Arnett, J. J. (2000). Emerging adulthood: A theory of development from the late teens through the twenties. American psychologist, 55(5), 469. DePaul University. (2013). Enrollment Facts. http://www.depaul.edu/emm/facts/ Fox, K. F. (2011). "Figuring it Out": A Grounded Theory of College to Post-College Transition. Fitzpatrick, J., Sanders, J., & Worthen, B. (2003). Program evaluation: Alternative approaches and practical guidelines (3rd Ed.) New York: Longman. Franke, R., Ruiz, S., Sharkness, J., DeAngelo, L., & Pryor, J. (2010). Findings from the 2009 administration of the college senior survey (CSS): national aggregates. Higher Education Research Institute at the University of California Los Angeles, Los Angeles, CA. Goodwin, L. L. (2006). Graduating class: Disadvantaged students crossing the bridge of higher education. Albany, NY: State University of New York Press. Hartel, W. C., Schwartz, S. W, Blume, S. D., & Gardner, J. N. (1994). Ready for the real world. Belmont, CA: Wadsworth Publishing Company. Olson, J. S. (2010). Chasing a Passion: The Early-career Lived Experience of First-generation College Graduates (Doctoral dissertation, The Pennsylvania State University). Nolan, L. J., & Jenkins, S. M. (2013). Transitioning Students out of College: The Senior LC in Psychology at Wagner College. New Directions for Teaching and Learning, 132, 132, 31-42. Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA: Jossey-Bass. Stokes, H., & Wyn, J. (2007). Constructing identities and making careers: Young peoples
30
perspectives on work and learning. International Journal of Lifelong Education, 26(3), 495-511. Walls, J. (2002). The senior experience: A transition to the world of work. Chicago, IL: Careers Across America Conference. (ERIC Document Reproduction Service No. ED465923) Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (Eds.). (2010). Handbook of practical program evaluation (Third Edition). San Francisco, CA: Jossey-Bass.
31
Appendix A: Logic Model
32
Appendix B: Pre-Program Survey Email Dear Student, The Office of Multicultural Student Success created a Senior Transition Program that will better equip its graduating seniors for post college life. By targeting graduating seniors involved in this post college success program the office hopes to alleviate some of the problems that students encounter as they prepare for post-college life. Your experience is important to us and we want to ensure that you are on your way to achieving crucial learning outcomes and goals. The answers you provide on this survey will help us determine if the program is successful at preparing you for your post college transition experiences and if it met its intended outcomes. *Those that complete the survey will be eligible to win a $5 gift-card to Starbuck's. Any questions related to this survey and the evaluation can be directed to Alicia DeLaRosa at delaros10@gmail.com or Reina Salcedo at reina.salcedo1@gmail.com Thanks, OMSS Staff
33
Appendix C: Pre-Program Consent Form& Survey
34
35
36
37
38
39
40
41
42
Appendix D: Pre-Program Survey Construct Map

Appendix B: Survey 1 Construction Map and Matrix
Senior Transition Pre-Program Survey (Time Point 1) Construction Map and Matrix
Logic Model Component CM Q# SQ# Question Answer Type Answer Options
Post Graduation
PPP PPP1.1
Question: After I 1 After I graduate from Depaul
Drop Box
( Yes, I plan on enrolling in graduate school.; No, I do not plan on enrolling in graduate school.; undecided; Yes, I plan on becoming employed.; No, I do not plan on becoming employed.; other plans)
Controlling for External Factors
CEF CEF 1.1 CEF 1.2 CEF 1.3 CEF 1.4 CEF 1.5 CEF 1.6
2 3 4 5 6
Please indicate your level of agreement with the following statements: I feel prepared for my life after DePaul I feel confident in designing a personal financial plan I have designed a personal financial plan I feel confident designing a career plan I have designed a career plan
Radio/Likert Scale (Strongly Agree to Strongly Disagree
*5 *4 *3 *2 *1
Checkbox Checkbox
Yes/No Yes/No
7 I feel confident in applying concepts of socially responsible leadership Socially responsible leadership takes into account the social, humanitarian and environmental consequences and strives to improve these aspects in any organization. Please answer the following questions Never/Rarely/Sometim es/Often/Very Often Not applicable
Addressing Assumptions
AA
AA1.1 AA1.1a
8 Please indicate the level of interaction you have had with OMSS staff: 9 *Purpose of Interaction
Checkbox Fill in Radio/Likert Scale (Strongly Agree to Strongly Disagree
AA 1.2
Please indicate your level of agreement with the following statements: 10 Overall, OMSS staff were able to answer my questions
*5 *4 *3 *2 *1
Hoping to gain from the program STOC 1.1 STOC1.2 STOC1.3 STOC 1.4 STOC1.5 Comfort with staff and faculty STOC2.1 STOC2.2 Plans STOC3.1 STOC3.1a STOC3.1b STOC3.2 11 12 13 14 15
Please indicate your level of agreement with the following statements: I want to learn to make personal financial management decisions I hope to improve my financial literacy I want to learn to identify career objectives I want to learn career mobility and success strategies I want to learn the concepts of Socially Responsible leadership
*5 *4 *3 *2 *1
Checkbox 19 I believe that DePaul faculty and staff care about my success 20 I hope OMSS faculty & staff maintain contact with me
Yes/No
27 I plan to: apply to Graduate School *If applicable, where? *If applicable, are of interest you plan to study? 28 I plan to: apply to Jobs
Checkbox Fill in Fill in Checkbox
STOC3.3 STOC3.4 STOC3.4a STOC3.5 STOC3.5a STOC3.6 Additional Questions STOC4.1 STOC4.2 STOC4.2a STOC4.2b General Information STOC5.1 STOC5.2 STOC5.3 STOC5.4 STOC5.5 STOC 5.6
29 I plan to apply to the following types of jobs 30 I plan to take some time off 31 32 33 34 If you plan to take some time off, please explain the reason why. I plan to travel If you plan to travel, please explain the reason why. Do you have other plans? If so, please describe them below.
Checkbox Checkbox Fill in Checkbox Fill in Fill in
Yes/No/Undecided Not applicable Not applicable Yes/No/Undecided Please check all that apply (Full-time, parttime, seasonal, temporary, other) Yes/No/Undecided Not applicable Yes/No/Undecided Not applicable Not applicable
35 36 37 38
Please answer the following questions What additional resources if any, do you think DePaul should offer to aide your post Depaul life transition? Fill in Are you involved in an additional senior success program? Checkbox If you are involved in an additional program, please list the name of the program(s). Fill in *What types of resources do they offer? Fill in Fill in
Not applicable Yes/No Not applicable Not applicable Not applicable
40 41 42 43 44 45
What is your What is your What is your What is your What is your What is your
university identification number? email address? gender? ethnicity? age? expected graduation date:
Checkbox Checkbox Fill in Date
Yes/No Yes/No Not applicable Fill in
43
Appendix E: Post-Program Survey Email Dear Student, The office of Multicultural Student Success created a Senior Transition program that will better equip its graduating seniors for post college life by targeting graduating seniors involved in post college success programs to help alleviate some of the problems students. Your experience is important to us and we want to ensure that you are on your way to achieving crucial learning outcomes and goals. The answers you provide on this survey will help us determine if the program is successful at preparing you for your post college transition experiences. *Those that complete the survey will be eligible to win a $5 gift-card to Starbuck's. *Those that complete the survey will be eligible to win a $5 gift-card to Starbuck's. Any questions related to this survey and the evaluation can be directed to Alicia DeLaRosa at delaros10@gmail.com or Reina Salcedo at reina.salcedo1@gmail.com Thanks, OMSS Faculty & Staff
44
Appendix F: Post-Program Consent Form& Survey
45
46
47
48
49
50
51
52
53
54
55
Appendix G: Post-Program Survey Construct Map

Appendix E: Survey 2 Construction Map and Matrix
Senior Transition Post-Program Survey (Time Point 2) Construction Map and Matrix
Logic Model Component CM Q# SQ# Question The following questions are in regards to your understanding prior to the participating in the Senior Transitions Program. The following questions are in regards to your understanding prior to the participating in the Senior Transitions Program. Prior to participating in the Senior Transitions Program, I received post1 college life transitioning support If applicable, please list from whom and what type of support you 2 received? Answer Type Radio/Likert Scale (Strongly Agree to Strongly Disagree) Answer Options
Prior Support
PS
*5 *4 *3 *2 *1
PS1.1 PS1.1a
Fill in
Not applicable
Post Graduation
PPP PPP1.1
Question: After I 3 After I graduate from Depaul
Drop Box
( Yes, I plan on enrolling in graduate school.; No, I do not plan on enrolling in graduate school.; undecided; Yes, I plan on becoming employed.; No, I do not plan on becoming employed.; other plans)
Follow Up STOC1.1 STOC1.2 STOC1.3 STOC1.4 STOC1.5 STOC1.6 STOC1.7 STOC1.8 4 5 6 7 8 9 10 11
STOC1.9 STOC1.10
12 13
Please indicate your level of agreement with the following statements: As a result of participating in the Senior Transitions Program, I feel prepared for my life after DePaul As a result of participating in the Senior Transitions Program, I feel confident in designing a personal financial plan As a result of participating in the Senior Transitions Program, I feel comfortable making personal financial management decisions As a result of participating in the Senior Transitions Program, I am able to demonstrate financial literacy As a result of participating in the Senior Transitions Program, I feel confident designing a personal career plan As a result of participating in the Senior Transitions Program, I feel comfortable identifying career objectives As a result of participating in the Senior Transitions Program, I feel comfortable identifying strategies for career mobility and success As a result of participating in the Senior Transitions Program, I feel confident in applying concepts of socially responsible leadership Socially responsible leadership takes into account the social, humanitarian and environmental consequences and strives to improve these aspects in any organization. As a result of participating in the Senior Transitions Program, I believe that OMSS staff care about my success
*5 *4 *3 *2 *1
STOC1.11 STOC1.12 Comfort with staff STOC2.1 STOC2.2 STOC2.3 STOC2.4 STOC2.5 Follow Up Questions STOC3.1 STOC3.1a STOC3.1b STOC3.2 STOC3.2a STOC3.3 STOC3.3a STOC3.3b STOC3.3c STOC3.3d STOC3.3e STOC3.3f STOC3.3g Additional STOC4.1 STOC4.1a STOC4.1b STOC4.2 STOC4.3 STOC4.4 General Information CM Q# STOC5.1 STOC5.2 STOC5.3 STOC5.4 STOC5.5 STOC5.6
As a result of participating in the Senior Transitions Program, I would feel 14 comfortable contacting a OMSS staff member to discuss my progress As a result of participating in the Senior Transitions Program, I plan to 15 maintain contact with an OMSS staff member Checkbox 16 17 18 19 20 Overall, OMSS staff were helpful resources. Overall, I felt comfortable learning from OMSS staff Overall, I felt comfortable seeking advice from OMSS staff Overall, OMSS staff presenters were engaging Overall, OMSS staff were able to answer my questions Yes/No
21 22 23 24 25 26 27 28 29 30 31 32 33
Have you applied to graduate school? If applicable, how many schools have you applied to? If applicable, please list the schools you have applied to. Have you been accepted into a graduate program? If applicable, how many schools have you been accepted to? Have you applied to jobs? If applicable, how many jobs have you applied to? If applicable, please list the positions you have applied for. Have you been asked to interview for a job? If applicable, how many job interviews have you had? Have you been offered a job position? If applicable, how many job offers have you recieved Please list the job offers you have recieved. Additional Questions Were you involved in any additional senior success programs? Fill in Checkbox Fill in Fill in Checkbox Fill in Not Applicable Yes/No Not applicable Not applicable Yes/No Not applicable
34 If you were involved in any additional senior transitions or success 35 programs, please list the name of the program(s). 36 If applicable, what types of resources did they offer? What additional resources, if any do you think DePaul should offer to aid 37 post-college life planning? Do you have any general comments in regards to the Senior Transistions 38 Program? 39 Would you reccomend this program to graduating seniors?
Fill in 40 41 42 43 44 45 What is your university identification number? What is your email address? What is your gender? What is your ethnicity? What is your age? What are your expected graduation date:
Not applicable
Checkbox Checkbox Fill in Date
See options See options Not applicable Fill in
56
Appendix H:
Participation Email Hello seniors, The Office of Multicultural Student Success (OMSS) staff would like to thank you for participating in the Senior TransitionProgram and also extend CONGRATULATIONS on your upcoming graduation. As we prepare to end the school year, we would like to invite you to participate in a focus group session offered by the OMSS staff to gain a better understanding of your overall experience within the Seniors Transition Program.
There will be two available sessions for you to attend: Tuesday, May 26th at 12pm at Lincoln Park Campus Student Center Room 312 Wednesday, May 27th at 12pm at Lincoln Park Campus Student Center Room 312 **Lunch and refreshments will be provided.**
Please RSVP via email with your session date in the subject box; by May 19th. Participants will receive a confirmation email as well as reminder. If you have any additional questions please contact Reina Salcedo at rsalcedo@luc.edu or Alicia DeLaRosa at adelarosa1@luc.edu.
Have a great week! (Signature)
57
Appendix I: Consent Form Consent to participate in the Senior Transition Program focus group Introduction: The focus group session findings are for institutional use only and will be utilized to better assess and make improvements to the Senior Transition Program. The information obtained from the focus group will remain confidential; information will only be shared with OMSS staff will. Demographic information is requested for data analysis purposes only. Please read this form carefully and ask any questions you may have before deciding whether to participate in this session or not. Purpose: The purpose is to gather information about students experiences during the senior transition program and use the information gathered to help make improvements and changes to the program for future students. Procedures: If you agree to participate you will be asked to respond to a series of questions. The session will last approximately 45 minutes but no more than 60 minutes. You are encouraged to respond openly and honestly to the questions. Risks and Benefits There are no known risks involved with participating in this study. Participation in this session is completely voluntary, if at any time you feel uncomfortable answering a question, you may remove yourself and or decide not to answer the question. Confidentiality: Your name will not be associated with your responses in the focus group reports. Personal information will be used solely to link responses to from previous survey responses. We will compile a report of basic themes and share it with you prior to finalizing it. Voluntary Participation: Participation in this focus group is voluntary. Even if you decide to participate, you are free to withdraw from participation at any time without penalty.
58
Contracts and Questions: Any questions related to this focus group can be directed to Reina Salcedo at rsalcedo@luc.edu or Alicia De La Rosa at adelarosa1@luc.edu. Statement of Consent: Your signature and demographic information below indicate that you have read and understood the information provided above, have had an opportunity to ask questions, and agree to participate in this focus group. You will be given a copy of this form to keep for your records.
Gender
______________
Ethnicity _______________
___________________________________________ Student Signature
____________________ Date
59
Appendix J: Focus Group Protocol Focus of the Focus Group: To gain a better understanding of the impact that the Senior Transition Program had on students. In addition, we would like to gauge whether the program met the overall outcomes it was intended to meet. Specific foci of the focus group: 1. Aspects of the program that made an impact on students. 2. Skills gained in order to enhance career capital. 3. Students level of preparedness for life after DePaul. Focus group script: Hi, my name is (INSERT NAME). Thank you all for choosing to participate in our focus group. The goal is to discuss your experiences with the Senior Transition Program and gain some information about what you felt was successful and what perhaps needs some improvement. As you recall, the purpose of the Senior Transition Program was to assist you in preparing for your post DePaul life by enhancing your career capital or graduate school knowledge. As you recall, you were asked to take a survey at the beginning of the Fall Quarter and then once again at the end of the Spring Quarter. The surveys you were asked to take were meant to provide a gauge of your skill and comfort level with post DePaul planning. The surveys have also helped us understand a little more about the Senior Transition Program and how to make it more successful and impactful for student participants. This session will take approximately between 30-45 minutes and is completely voluntary. There might be instances where I ask for more clarification or details but I would like you to keep in mind that you are entitled to share as much as you feel comfortable with as well as choose to not answer certain questions. With that being said, I would like to remind you that all of the information you share during this focus group is confidential and will only be used to help improve the Senior Transition Program; therefore information will only be seen by the OMSS Staff. Before we begin I ask that you please read the focus group consent form carefully and if you have any questions please let me know before you sign. By signing the consent form you are agreeing to participate in this focus group. Introductory Question: To start off, can you please tell me how you learned about the Senior Transition Program? o Probes:
60
What made you decide to join the program? What did you hope to gain from the program?
Foci Question 1-2: Aspects of the program that had an impact on students. o What part of the program (if any) did you feel was most valuable? o Probes: Why? Talk about what you gained from this particular program activity. Was there anything that you felt could have been different to make this aspect of the program better? o What part of the program (if any) did you feel was the least valuable? Why? What did you feel could have different to make this aspect of the program better? Foci Question 3-5: Skills gained in order to enhance career capital. What do you feel you have gained from this program that could be helpful in your career planning (if anything)? o Probes: What (if anything) would you have hoped to gain that you did not from the program? Has the program enhanced your financial literacy? (Why or Why not?) o Probes: Do you feel that you are prepared to design a personal financial plan? Has the program enhanced your leadership efficacy? (Why of Why not?) o Probes: Are you able to apply concepts of social responsibility to your everyday life? (Why or Why not?)
Foci Question 6: Students level of preparedness for life after DePaul. How has the program prepared you for life after DePaul (if it has)? o Probes: How can the program enhance, if it is enhancing, the level of preparedness for you? Do you feel that you are prepared for once you leave DePaul?
Conclusion Throughout your participation in the program, were there any experiences you felt were crucial in preparation for graduation? How has this experience helped you as you think of your career or next steps?
61
Is there any additional comments or information that you would like to share that has not been covered?
I would like to thank you for taking time out of your schedule to participate in this focus group. The feedback that you have provided will help in improving the program for future students. If you think of anything else that you feel could help in the improvement of the Senior Transition Program, please feel free to contact us. Have a great day and congratulations on your upcoming graduation.
62
Appendix K: Focus Group Construct Map

Construct Code SKILLS Meaning Skills Theme Skills students learned or wished they had learned in the program. What students felt they learned throughout the duration of program? What aspects of the program were of most value to students? How prepared and whether their preparedness has changed from the beginning of the program. Relationship between program and graduation. Comfort as graduation approaches. Ways in which program has affected their career choices.
LEARN
Learning experiences
VALUE
Most valued aspect(s) of program Level of preparedness
PREP
GRAD
Graduation
CAREER
Career choices
63
Appendix L: Budgets Senior Programs

Senior Retreat
Projected Cost Actual Cost Difference
Retreat Center Snacks Supplies
$3,512 $100 $100 $3,712

Projected Cost Actual Cost
$0
$3,512 $100 $100 $3,712

Difference
Subtotals
Senior Soiree
$0
Food stoles Band Programs
$2,275 $1,200 $400 $90 $3,965 $0
$2,275 $1,200 $400 $90 $3,965
Subtotals
Evaluation
Items $5 gift card to Starbucks Food Printing Recorder Foot Pedal Miscellaneous
Quantity 20 2 50 1 1
Price per Item $5 $100 $1 $100 $100
Cost $100 $200 $50 $100 $100 $100
64
Appendix M: Timeline Pre evaluation steps Development of Survey Design Development of qualitative protocol - focus groups Present evaluation plan to stakeholders Make adjustments as requested by stakeholders Present final evaluation plan to stakeholders Pre-test Preparation and Execution Invite students to complete pre-test survey Close pre-test Collect data from pre-test Analyze data collected from pre-test Report data to all stakeholders Post-test Preparation and Execution Meet with OMSS staff in planning for post-test and focus groups Refine questions for post-test and focus groups Post-test survey distribution Close post-test Analyze data collected from post-test Focus Group Preparation and Execution Continue to recruit potential focus groups participants Trial run of focus group protocol with OMSS staff Facilitate focus groups Transcribe focus groups data Analyze and report data to OMSS staff and respective stakeholders Next Steps Submit recommendations for improvements 2nd implementation for new senior co-hort Color Key Oct-13 Nov-13 Jan-14 Feb-14 Apr-14
Sep-14
Oct-14 Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May15
May15
Jun-15
Jul-15 Aug-15
Aug-15
Sep-15
2013
2014
2015
65
Appendix N: Poster Presentation
DePaul University Chicago

Office of Multicultural Student Success Senior Transitions Program Evaluation
Background
At the end of their college experience, seniors embark on a crossroad that challenges them to deal with a sudden shift in their identity College graduates face anxiety as they approach this impending transition process (Olsen, 2010) Students feel unprepared and need to learn about a wide range of life topics because most lack essential skills to succeed
Alicia A DeLaRosa & Reina Salcedo
66
Senior Transitions Program

Provides a series of capstone experiences focused on successful college to post-college transition Goal of the program is to successfully prepare graduating seniors for post-college success Target Population: graduating seniors that identify as lowincome, first-generation, and students of color Program Content Area:
Alumni Engagement and Participation Senior Recognition Events Career Action Planning and Attainment College to Post-College Transition Development
Evaluation Purpose
This is the pilot year of the program Aim is to better assess the effectiveness and depth that the Senior Transitions Program prepares seniors as they move into their post-college life successfully Ensure that the learning outcomes and program goals are feasible and attainable
Logic Model
Appendix A
SENIOR TRANSITIONS PROGRAM LOGIC MODEL
Inputs Activities
-Senior Retreat: Held in Spring quarter for students to come together -Career Coaching: One on one coaching done with an assigned OMSS staff member throughout the academic year. -Alumni Connections Workshop Series: Connect with alumni through workshops and panel discussions -Senior Soiree: Annual recognition for graduating seniors and their families and friends. Company Field Trips
Evaluation Approach
Outcomes Medium Long
Students will graduate with a higher sense of what career capital is and how to obtain it. Students will be able to budget their financial obligations such as student loans. Develop 5 year strategic plan for the continuous development of program. Previous students will continue active participation as alumni with the program. Develop a clear and quality pipeline between other programs in the office (STARS, PATH, MOC, WE) to ease the transition for students into the program.
Outputs Participation
Senior students
Short
Engagement from students, alumni and campus and community partners Students will have obtained a better understanding of expectations for life after DePaul
-Professional Staff -Alumni Participation -Time commitment -Money -Literature on post college success -Educational materials -Technology -Campus partners -Transportation -Community partners
Formative &Summative
Improve program effectiveness and success rate of students
OMSS Staff
DePaul Alumni
Process
How the program is structured to benefit the students
Campus partners: Career Center Alumni Relations Financial Aid Community Partners: These include businesses and company such as LinkedIn that students will visit.
Students will secure an internship during their senior year. Students will develop a resume and cover letter.
Students will be able to discern about career and professional decisions and opportunities.
Outcome
Have the students learned the appropriate information and tools to ensure that they have a successful post-college life transition
Assumptions
All students and alumni will be fully engaged. Campus and community partners will be able to dedicate enough time for the success of the program. Program will
External Factors
Budget in regards to monies allocated yearly for program. Students development on career capital will vary. Enrollment and retention rates for senior students.
Quantitative Approach
Census Sampling Pre-Survey through Google Forms
Incentive ($5 Starbucks gift certificate)
Qualitative Approach
Focus Group Two sessions after the quantitative data has been analyzed Convenience Sampling Participants Volunteer basis Two groups of 8 to 10 participants 30% (20/87) of overall participants Protocol Knowledge acquired about career goals and plans Knowledge acquired about financial literacy Knowledge acquired about leadership efficacy Opinions/beliefs about the program and specific workshops Analysis Construct Map/Coding
Post-Survey provided
Email invite Incentive ($5 Starbucks gift certificate) 17-20% response rate
Statistical Analysis
Multi-Regression Factor Analysis Cross tabs
67
A Priori Construct Map

Construct Code SKILLS Meaning Skills Theme Skills students learned or wished they had learned in the program. What students felt they learned throughout the duration of program. What aspects of the program were of most value to students? How prepared and whether their preparedness has changed from the beginning of the program. Relationship between program and graduation. Comfort as graduation approaches. Ways in which program has affected their career choices. LEARN Learning experiences
TIMELINE
Color Key Pre evaluation steps Development of Survey Design Development of qualitative protocol - focus groups Present evaluation plan to stakeholders Make adjustments as requested by stakeholders Present final evaluation plan to stakeholders Pre-test Preparation and Execution Invite students to complete pre-test survey Close pre-test Collect data from pre-test Analyze data collected from pre-test Report data to all stakeholders Post-test Preparation and Execution Meet with OMSS staff in planning for post-test and focus groups Refine questions for post-test and focus groups Post-test survey distribution Close post-test Analyze data collected from post-test Focus Group Preparation and Execution Continue to recruit potential focus groups participants Trial run of focus group protocol with OMSS staff Facilitate focus groups Transcribe focus groups data Analyze and report data to OMSS staff and respective stakeholders Next Steps Submit recommendations for improvements 2nd implentation for new senior co-hort 2013 Oct-13 Nov-13 2014 Jan-14 Feb-14 2015 Apr-14
Sep-14
Oct-14
Nov-14
Dec-14
VALUE
Most valued aspect(s) of program Level of preparedness
PREP
Jan-15
Feb-15
Mar-15
Apr-15
May-15
GRAD
Graduation
May-15
Jun-15
Jul-15
Aug-15
CAREER
Career choices
Aug-15
Sep-15
Budget
Items $5 gift card to Starbucks Food Printing Recorder Foot Pedal Miscellaneous Quantity 20 2 50 1 1 Price per Item $5 $100 $1 $100 $100 Cost $100 $200 $50 $100 $100 $100
Next Steps
Submit recommendations for program improvement 2nd implementation for new senior co-hort Piloting survey design and focus group questions to students Match comparison and sample t-test on demographic factors (race, gender, 1st gen) Control group in place Creating a strategic 5 year assessment plan for program

Running Head: Senior Transitionprogram Evaluation 1

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Running Head: Senior Transitionprogram Evaluation 1

Hochgeladen von

Copyright:

Verfügbare Formate

Running head: SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

learning outcomes aimed to achieve student success.

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

Appendix A: Logic Model

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

Appendix C: Pre-Program Consent Form& Survey

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

Appendix D: Pre-Program Survey Construct Map

Question: After I 1 After I graduate from Depaul

Controlling for External Factors

Radio/Likert Scale (Strongly Agree to Strongly Disagree

Checkbox Fill in Radio/Likert Scale (Strongly Agree to Strongly Disagree

Radio/Likert Scale (Strongly Agree to Strongly Disagree

Checkbox Fill in Fill in Checkbox

Checkbox Checkbox Fill in Checkbox Fill in Fill in

Not applicable Yes/No Not applicable Not applicable Not applicable

Checkbox Checkbox Fill in Date

Yes/No Yes/No Not applicable Fill in

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

Appendix F: Post-Program Consent Form& Survey

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION

SENIOR TRANSITIONPROGRAM EVALUATION