Running head: EVALUATION PLAN FOR LLC 1

Evaluation Plan for Leadership for Social Change Learning Community

Emma Hindman, Candice Germany, and Dana Gramuglia

Loyola University Chicago

EVALUATION PLAN FOR LLC 2

Table of Contents

Program Context 3
Logic Model Description 4
Statement of Problem 4
Theoretical Framework 5
Program and Evaluation Stakeholders 6
Quantitative Approach 7
Research Design and Method 8
Survey Instrument 9
Population and Sampling Frame 10
Statistical Analysis 11
Results Presentation 11
Qualitative Approach 11
Focus Group Participants 12
Focus Group Procedures 13
Implementation Plan 14
Moderators 14
Scheduling 15
Data Recording 15
Analysis Plan 16
Coding and Procedures 16
Validity and Reliability 17
Limitations 18
Final Report 19
Next Steps 20
References 21
Appendices 22
Appendix A: Staff and Faculty involved in LLC 2016-2017 22
Appendix B: LLC Monthly Meeting Topics 23
Appendix C: Required Academic Courses for LLC 24
Appendix D: Logic Model 25
Appendix E: Invitation to Participate in Survey 26
Appendix F: Pre-Test Assessment 27
Appendix G: Post-Test Assessment 29
Appendix H: Invitations to Participate in Focus Group 31
Appendix I: Consent to Participate in Focus Group 32
Appendix J: Focus Group Questions 34
Appendix K: Budget 36
EVALUATION PLAN FOR LLC 3

Program Context

Loyola University Chicagos Leadership for Social Change Learning Community (LLC)

serves first-year students who are seeking opportunities to explore introductory topics of

leadership, enhance their personal skills as leaders, and apply this knowledge through

engagement during their time at Loyola and beyond. The LLC began in the 2012-2013 academic

year in response to other institutions success and research in housing their own leadership

learning communities. (S. Howes, personal communication, Sept. 24, 2016). In the LLCs first

year, 64 first-year students were a part of the community, living in Regis Hall. Today, the LLC is

housed in Mertz Hall (1125 W Loyola Ave) with 80 students participating.

The LLC is overseen by multiple areas, including: the Office of Student Leadership

Development, the Department of Residence Life, and Academic Advising. See Appendix A to

see the names of individuals directly involved in the 2016-17 LLC.

There is currently no application process for students to be a part of the LLC, instead,

students are placed by preference in their housing assignment application for the upcoming year.

Students are placed on a first-come, first-served basis. (S. Siner, personal communication, Sept

20, 2016). Students in the LLC live together in Loyolas residence halls (this year, floors 3 and 4

of Mertz Hall) for the full academic year, attend required monthly meetings focused on different

topics surrounding leadership (see Appendix B), and take academic courses together (see

Appendix C).

Through a combination of living together in a residential community, monthly meetings

and programming, and taking academic courses together, the LLC seeks to develop a sense of

community and connection with each other, as well as with the student leaders, staff, and faculty

supporting the learning community. The program seeks to help students feel empowered and
EVALUATION PLAN FOR LLC 4

capable as they engage on-campus and beyond, gain knowledge, skills, and attitudes that

contribute to their development as socially responsible leaders. More specifically, the LLC

program allows opportunities for students to practice working collaboratively for positive

change, achieve academic success, and apply leadership principles in their academic pursuits

(leadership is interdisciplinary in nature), develop an increased understanding of self and others

with regard to socially constructed identities as they relate to leadership, power and privilege,

and social change further develop their knowledge and skills related to advocacy. (S. Howes,

personal communication, Sept. 28, 2016).

Logic Model Description

To help develop this evaluation plan, a logic model was used (see Appendix D) The

graphic below is a representation that consists of the input, outputs, and outcomes of the learning

community. The inputs for this logic model that are most important to recognize are personnel,

time, and financial constraints. The outputs for this logic model, which are a result of the inputs,

are arranged into activities and products. Through the outputs, the short-term, medium-term, and

long-term outcomes are then described for the Leadership for Social Change Learning

Community.

Statement of Problem

Since the initial implementation of the LLC at Loyola University Chicago, there have

been a significant amount of changes to the program. Staff and faculty working in and with the

LLCs has changed, the floor has moved buildings into a different environment, and the number

of students in the program has fluctuated. There is a new Assistant Director of Academic

Support and Learning Communities, Sam Siner, overseeing the LLC and they are now located in
EVALUATION PLAN FOR LLC 5

a different hall. The number students have dramatically shifted through the years. Currently

there are about eighty residents of the LLC, which is average in comparison to previous years.

This project will be used to evaluate the student engagement of the students living in the

Leadership for Social Change Learning Community. This evaluation will be used by the

evaluators to determine the extent to which the students of the Leadership for Social Change

Learning Community are engaging with, or building community with each other, the university,

and the greater Chicago community. This will be a formative approach to the evaluation process

of the LLC. Because of the number of significant changes the LLC has gone under, only one

evaluation has ever been done on the community. This evaluation was given the first year of the

LLC inception and there has not been one done in the last three years.

Theoretical Framework

In the developing our evaluation plan, it is important to consider research and literature

that serve as a foundation for better understanding the student population and programming that

the LLC provides. Evans and Reason guiding principles for student affairs reflects the intention

behind the LLC: 1) A focus on students as a primary purpose of our work 2) Recognition of the

role of environment in a students experience. 3) Intentional and empirically grounded in practice

and theory, 4) Reflects a responsibility to broader society. 5) Social justice advocacy. (2011, p.

370-376) The LLC is driven by these intentions, providing a holistic experience through

different learning environments that challenges students to understanding leadership as a

responsibility to enact positive social change.

Research has shown the value of college campuses extending learning beyond of the

classroom environment, and that learning communities and collaborative learning should become

a norm within campuses (Tinto, 1999, p. 6-7). Through learning communities, institutions are
EVALUATION PLAN FOR LLC 6

able to do a better job to enable students to bridge the divide between academic classes and

the social construct that frequently characterizes student life. Students tend to learn and make

friends at the same time, and as they spend more time together learning, they learn more. (Tinto,

1999, p. 8)

Particularly within the first-year college experience, Tinto speaks to five conditions that

are essential to college retention: expectations, support, feedback, involvement, and learning

(Tinto, 1999, p. 2). Through the LLC providing clear expectations of requirements to students,

peer-mentorship through Resident Assistants, opportunities for continuous involvement within

their first-year, and learning both in and outside of the classroom, the LLC can be seen as an

incredibly valuable part in the success of the student, but also in their retention.

Program and Evaluation Stakeholders

Through an analysis we identified several individuals and groups as stakeholders within

the Leadership for Social Change Learning Community (LLC). The stakeholder analysis

organizes those identified into four different quadrants based on their power differential and

interest level. The stakeholders in the top left quadrant have the high interest and low power- are

identified as the subjects (Bryson & Patton, 2010). Within the context of the LLC the subjects

are those living in the learning community. These subjects have high interest in the LLC due to

their involvement, but low power due to their lack of positional authority. The subjects will be

the ones most impacted by any changes made due if the observations through the evaluation

were to be implemented. The opposite quadrant, are identified as the players and have high

power and high interest.

The players for the LLC are: Sam Siner,Director of Academic Support and Learning

Communities, Shannon Howes, Director of Student Leadership Development, and Marilyn

EVALUATION PLAN FOR LLC 7

Krough, Faculty Director of Learning Communities, Academic Advising. Due to their direct

involvement with the LLC and connection to the students involved they have high interest in the

program. Because of the positional authority they hold within the university their level of power

is identified as high. This high power paired with high interest allows them to advocate for the

LLC and the students in the learning community. The players in this context will be in the

position to make changes if any are warranted after the evaluation is complete. Due to their

unique position of holding both power and interest constant we thought it be best to consult the

players throughout the evaluation process. Consultations were held with the players of the LLC

to yield more information and to gain a full sense of their role in the learning community.

The lower right quadrant houses the context setters, who have high power and low

interest. The context setters at Loyola University Chicagos Leadership for Social Change

Learning Community are: Deb Schmidt Rogers, Director of Residence Life, K.C. Mmeje,

Assistant V.P. and Dean of Students, Jane Nuefeld, Vice President of Student Development.

These individuals hold the most power of those in listed in the matrix due to their position at the

university. This position of power and authority allows them to make policy changes and

decisions which affects how the learning community operates. However, this same position can

create a gap between themselves and the students which their decisions impact.

Finally, the lower left quadrant houses the crowd, which has the least power and interest.

In the context of the LLC, the crowd is the general student body at the university- those who are

not involved with the learning community. The general student body was designated as the

crowd in relation to the LLC due to their lack of power within this context. The LLC also does

not directly impact the general student body- they have little to no stock in the learning
 EVALUATION PLAN FOR LLC 8

community. The Power versus Interest grid (Bryson & Patton, 2010) can be found below with all

of the stakeholders listed in their respective quadrants.

Subjects (High Interest, Low Power)
Students living in the Leadership Learning
Community
Resident Assistants for Leadership Learning
Community
Players (High Power, High Interest)
Sam Siner, Assistant Director of Academic
Support and Learning Communities
Shannon Howes, Director of Student
Leadership Development
Marilyn Krough, Faculty Director of Learning
Communities, Academic Advising
Resident Director for Leadership Learning
Community

The Crowd (Low Power, Low Interest)
General student body (Students who do not
live in the Leadership Learning Community)
Context Setters (High Power, Low Interest)
Deb Schmidt Rogers, Director of Residence
Life
K.C. Mmeje, Assistant V.P. & Dean of
Students
Jane Nuefeld, Vice President of Student
Development

Quantitative Approach

For the purpose of this assessment, we will be conducting mixed methods research to

address our evaluation questions. These questions will hone in on the students learning of

leadership in the learning community. Specifically discussed in this section will be the

quantitative approach within the assessment. The quantitative approach will be the primary

focus of the mixed methods assessment, and will produce the majority of the data compared to

the qualitative approach. The sections below will focus on the research method and design,

survey instrument, population and sampling frame, statistical analysis and the results

presentation.

Research Design and Method

In order to effectively assess and seek opportunities for improvement within the

Leadership Learning Community, we made a collaborative decision with stakeholder Sam Siner,
EVALUATION PLAN FOR LLC 9

Assistant Director for Academic Support and Learning Communities, to focus on the learning

outcomes of the program through a one-group pre-experimental pre-test-post-test design

(Creswell, 2009). This is a described as a pre-test measure. Then there is a treatment of the

participants, which is their education of leadership. Finally, there is the post-test measure

implemented to measure the learning that took place during the treatment. Since we will be

sampling on group at a time, a census sampling will be used because of the manageable size of

the LLC cohort (80 students in 2016-2017 academic year) and will distribute and require all

students to take part during both the first and last required meetings of the year.

Survey Instrument

The link to the pre-test survey (Appendix F) will be given to the group of participants and

they will be given notice that they must complete a survey during their first required monthly

LLC meeting on the floor that the LLC resides in. The assessment will include nine Likert-scale

questions broken into three areas within the LLCs learning goals: campus engagement,

understanding of social identity, and advocacy.

The first category is designed to measure a students sense of empowerment to engage

with Loyola staff and faculty, fellow LLC students, and non-LLC students. Students will select

their comfortability on a five-point scale, with 1 being not at all and 5 being very. The

second category is designed to measure a students understanding of social identity in relation to

leadership, power and privilege, and social change. Students will select their understanding on a

five-point scale, with 1 being strongly disagree and 5 being strongly agree. The third and

last category is designed to measure a students willingness to engage in conversation and

activities related to advocacy. Students will select their willingness to engage on a five-point

scale, with 1 being strongly disagree and 5 being strongly agree. The survey should take no
EVALUATION PLAN FOR LLC 10

longer than ten minutes to complete. Following this, two demographic questions regarding

gender and race will be included. This allows the stakeholders to hold data for future research.

This will also allow them to see if there are groups of individuals not receiving the same learning

experiences or opportunities.

Students will be given the survey through Orgsync, also known as LUCentral on campus.

Orgsync is a system that they university has invested in and can be utilized in many different

ways. It is an online system that allows users to access it from anywhere. OrgSync allows for

data to be collected confidentially. OrgSync also keeps the data from each participant together

without identifying the participant. Both the pre-test and the post-test will be administered

through Orgsync.

The post-assessment (Appendix G) is a reflection of the pre-assessment, and the link will

be given at the last required monthly LLC meeting of the year, which takes place during the first

week in April. The date and time change each according to the schedule of Shannon Howes, the

Director of Student Leadership Development, who leads the meetings.

Population and Sampling Frame

Because the population is so small-in the academic year 2016-2017 there are 80 students

enrolled, we will be wanting to utilize the entire population for the pre-assessment and post-

assessment survey. Utilizing the entire population would be difficult, because not all participants

will want to participate. For this survey to have statistical power a group of 60-65 students

would be sufficient. This will give an accurate depiction of each students growth after

completing the program, and allow us to see the impact on the whole population as well. There

will be a sample of this population of first year students utilized for the purpose of the focus

group.
EVALUATION PLAN FOR LLC 11

Statistical Analysis

A paired samples t-test would be best used to analyze the data. This would be used to

view the means between sections beginning and end experiences. The use of a paired samples t-

test will allow researchers to compare results of two different tests. In this case, the pre-test and

post-test results will be compared. The analysis of the data will happen with the use of the

Statistical Package for the Social Sciences (SPSS), provided by Loyola University Chicagos

School of Education.

Results Presentation

The assessments findings will be compiled and shared with the primary stakeholders of

the Leadership Learning Community, Sam Siner, Shannon Howes, and Marilyn Krough. The

results of the quantitative survey will be combined with the qualitative pieces of the survey

during the presentation. There will be documentation presented from the pre-test and the post-

test in bar graphs for visual representation. There will also be a written narrative that compares

the data. The findings from the qualitative part of the study will also be included in the

presentation. This will be used to support the quantitative data.

Qualitative Approach

For the purpose of measuring the learning outcomes for the Leadership for Social Change

Learning Community at Loyola University Chicago, we will be conducting mixed methods

assessment. Specifically discussed in this section will be the qualitative approach within the

assessment. There will be a focus group, which will serve as the qualitative approach within this

mixed methods assessment. The purpose of the focus group will be to garner further insight of

the data collected through the quantitative assessment. In other words, while the quantitative

portion gave a hard data set through numbers, the focus group will attach meaning through words
EVALUATION PLAN FOR LLC 12

and responses given during the assessment. This will help when assessing the whole data set, to

determine how successful the learning outcomes of the LLC are over the course of the academic

year.

Focus Group Participants

Due to the inability to utilize all 80 students involved in the program during the focus

group portion we will need to identify a representative sample. Those who can participate in the

focus group will be limited by two determining factors: individuals who have completed the

Leadership for Social Change Learning Community [one full academic year] and have

participated in the quantitative portion of the study pre and post-assessment survey. The

individuals who have completed one year of the learning community and the pre and post-

surveys will receive an invitation to sign up for the focus group. There will be a deadline to

communicate interest in the focus group due to the short period the academic year provides

access to the students. The participants will not receive a tangible item as an incentive- rather

they will be given gratitude through gestures such as thank you cards. For example, the students

have been informed of the impact their participation in the assessment process will have on the

experience of the upcoming learning community.

There will be two focus groups, in order to capture a large enough sample group of the

population. This will also allow smaller groups, so each participant can contribute equally to the

information gathered through the qualitative assessment. Both of the focus groups will be

administered the same questions and given by the same proctors. In order to gain the best

perspective of growth in the efficacy of the learning outcomes, the focus group will take place at

the end of the academic year- once all expectations of the participants has been completed.

Focus Group Procedures

EVALUATION PLAN FOR LLC 13

The focus group protocol (Appendix J) will be strictly followed when proctoring the

focus group[s]. There will be seven open ended questions which the proctor will follow, a

scripted opening and closing, and a space for closing comments. The structure of the focus

group will be outlined in detail below.

The focus group participants will be brought into the room where the focus group will be

conducted. The focus group participants will have time to get settled and find their desired seat-

the seating arrangement will be chairs in a large discussion circle to encourage an honest, open

dialogue about the individuals experiences throughout the learning community experience.

Once the moderator has assessed the participants are settled, the introduction to the goals and

learning outcomes of the focus group will be introduced. The participants will receive an

informed consent form (Appendix I) once an overview of the program has been communicated.

The consent form allows for the following conditions: if you feel uncomfortable you may leave

at any time, to not answer questions posed directly or indirectly, and the protection of

confidentiality. The signing of this form will allow us [researchers], to ethically include the

results of the focus group into the final report. After the forms have been completed and

collected, the questions can be administered to the focus group.

The focus of the focus group, is to measure - through the questions asked-the extent to

which the learning outcomes of the Leadership for Social Change Learning Community are

being met. While the questions will be answered, it is up to the moderator to set the tone for the

space of the focus group. This will include being a gatekeeper of the frequency of information

from each individual; allowing a space for each individual to become equally involved in the

focus group. The gatekeeping of the conversation will be crucial to the end resulting data- if one

or two people dominate the conversation there will not be a representative sample of the
EVALUATION PLAN FOR LLC 14

population reflected in the assessment. The moderator will also supply clarification of any

questions if so asked by the participants. Once all of the questions have been administered by

the moderator and answered in depth, there will be space provided for final comments. After no

more than twenty minutes has been given for the final comments [or all are given], the moderator

with supply a concluding remark [through the provided script]. The moderator will be prompted

at this time to give thanks for the participants time and again to reiterate the objective of the

focus group and how their contribution allowed for this to be fulfilled.

Implementation Plan

The main idea for the focus group is to make the participants comfortable enough to be

able to share willingly with the researchers. The use of moderators, scheduling of focus groups

and recording of group discussions are all extremely necessary to create an environment that

participants are willing to talk about their experiences.

Moderators

Moderators of the focus group can have a significant impact on the results of the focus

group. The moderator of the focus groups will be Candice Germany, who is a co-contributor of

this evaluation. Candice is currently an intern in Student Leadership Development at Loyola

University Chicago and is supervised by Shannon Howes, so she has knowledge of the

Leadership Learning Community and their requirements. Candice does not have direct contact

with students that currently live in that community. Emma Hindman and Dana Gramuglia, also

co-contributors to this evaluation plan, will act as the assistant moderators during the focus

groups. Dana currently has no direct contact with community, other than the information given

to her during this evaluation process. Emma works as an Assistant Residence Director in the

same residence hall where the learning community resides. The assistant moderators will handle
EVALUATION PLAN FOR LLC 15

logistics of the focus group. Their main responsibilities are checking to make sure the

participants are comfortable and all equipment is working effectively. Having assistant

moderators will lessen the amount of personal bias the moderator may have that could impact the

focus group. Personal bias of the moderator will be controlled by having the presence of an

assistant moderator; the personal bias is controlled by having a representative of another

perspective in the room.

Scheduling

Focus group implementation will take place in mid April- at the end of the academic

term. This timeline was created with the undergraduate students in mind- who are usually

preoccupied with end of the year papers and final exams during the month of May. The date of

the focus group will be set the week before fall break- early October; by scheduling this far in

advance we hope to avoid conflicts with other events and commitments. The scheduling of the

focus group will be communicated in advance to the students involved through email. There will

be a conference room scheduled in Damen Student Center, located within a short walking

distance of Mertz Hall- where the learning community resides- to make it convenient for those

involved in the focus group.

Data Recording

While the focus group takes place, there will be two different methods of capturing data.

Dana and Emma will both be responsible for taking complete notes during the focus group. The

observations being noted will include (but not limited to): body language, voice inflections, and

the spoken content. The goal behind utilizing two note takers will be two allow the researchers

to minimize personal bias. The second method of capturing data will be audio recording. The

method of using audio recording was preferred over recording using video equipment in order to
EVALUATION PLAN FOR LLC 16

preserve the integrity of the confidentiality communicated to the participants. With the concept

of confidentiality in mind, there will be procedures within the transcribing process to preserve

the identity of those in the focus group. The audio recording will then be transcribed and coded

by Emma and Dana; who have already witnessed the identity of the focus group participants.

The purpose behind only allowing Emma and Dana to listen to the audio recording is to prevent

anyone involved with the learning community to identify a focus group participant by their voice

or speech patterns. The other purpose of providing a transcription and coding of the focus group

is make the results of the focus group accessible to the researchers and allow them to make

conclusions based on the data provided.

Analysis Plan

The focus group will be structured around the learning that has potentially occurred and

the student engagement in leadership on campus. The coding of the data will be focused on

recognizing those statements made by the focus group. Anticipated codes would be: leadership

skills, community involvement, and social justice. Another part of the assessment is to find other

ways to get students to engage in learning around leadership and other ways to encourage student

engagement in leadership. These will be coded as well.

Coding Procedures

Though there are many types of coding, descriptive coding will be used during this

process as it was described by Rogers and Goodrick (2010). The transcribed document will be

uploaded to Microsoft Word, and each code will be assigned a highlight color. Dana, Emma, and

Candice will go through the transcribed document and highlight each code in its required color.

Gathering and evaluating the data will take a significant amount of time.
EVALUATION PLAN FOR LLC 17

In the coding process following the focus group, coders will follow processes outlined by

Rogers and Goodrick (2010) in which they will individually read through and identify common

themes and differences between dialogues, then forming definitions for each unique code, which

is also known as pattern coding. After coders have worked through transcripts separately, they

will then come together and compare their results to ensure that they are being consistent in

identifying themes and important focuses.

Validity and Reliability

In an effort to ensure that the results of coding from our focus group interviews are valid

and reliable, we plan on taking steps to ensure coders as well as moderators leading focus groups

will not have a direct relationship with any of the stakeholders and departments involved in the

Learning Leadership Community. Coders will use member checking throughout the process as

described by Krueger and Casey (2010) where moderators will occasionally repeat back what

has been shared and summarize common themes that came up in dialogue back to the

participants, confirming that these topics were of importance and relevant to the Learning

Leadership for Social Change Learning Community.

When focusing on validity and reliability there are several protocols which can be

established to ensure the assessment process is true and valid. According to Schuh (2009),

triangulation is one of those protocols, using multiple sources of data, data collection methods,

or both, and multiple investigators to collect data (168). There are multiple sources to ensure

triangulation takes place: theoretical sources, those provided by LLC, and the information

gathered from the pre and post-test as well as the focus group.

When establishing validity and reliability, the identities of the moderators and the impact

it creates should be recognized before conducting the focus groups. The ways in which the
EVALUATION PLAN FOR LLC 18

identities can impact the space of the focus group are as follows: our own perceptions when

taking down the observations of the focus group (e.g., body language, voice inflections, and the

spoken content), the way others present themselves in the focus group, and the answers provided

in the focus group by the participants. These different attributes based on our identities can

affect the validity and reliability of the data. The identities we, as moderators are bringing into

the space are white, females.

Limitations

The main concern and potential limitation that may come from the focus group is the fact

that individuals involved in the focus group all have a personal relationship with one another

through the program, classmates, and living with one another in their shared community within

the residence halls. As a result, there is a higher potential for conflict between individuals and

outside biases that may influence participants willingness to share and what they choose to share

or not share in the space.

Secondly, it is incredibly important that the moderator work to remain open-minded and

create a space where all participants are able to share freely. This includes setting expectations

and conditions for success at the beginning of the meeting with the group that the dialogue

should focus around the learning experience within the learning community and that this is not a

space that is meant to target others, but rather a time to speak to successes and opportunities for

growth within the program itself.

Finally, because the space is filled with self-identified leaders, it is likely that many

participants will be eager to share their opinions and experiences. It is essential that the

moderator is providing opportunities for all students to participate in the conversation and is

stepping in to redirect conversation to others if one individual is dominating the space. By

EVALUATION PLAN FOR LLC 19

ensuring all participants are sharing, researchers are better able to avoid generalizing findings to

the experiences of the cohort to only a few students.

Final Report

The resulting themes found in coding will be compiled and presented as a narrative report

by the coders in collaboration with researchers Emma Hindman, Candice Germany and Dana

Gramuglia. The report will note the consistent themes identified by coders in addition to

outlying themes that are of importance, but not as frequently referenced by participants

throughout the focus group sessions. A concept map will be used as a visual aid to better

understand the connections and relationships between themes and the flow of the focus group

dialogue. The final report will be presented for review to major stakeholders within each

department: Academic Advising, Residence Life, and Student Leadership Development

Next Steps

Once the assessment portion is completed there are several steps we have identified

which will need to occur annually along with the program. Due to all of the current researchers

graduating in May there will need to be a new set researchers identified. The new set of

researchers will help ensure the evaluation and assessment process will occur every academic

year. Until this assessment, there has not been a regular and consistent method of evaluation,

thus not allowing for a true comparison of results of learning outcome execution from year to

year. With the process set in place and the new set of researchers this will be implemented each

academic year. With the consistent assessment and implementation of the evaluation there will

need to be an evaluation of trends- this will be identified through the coding of the focus groups

and and the results of the pre and post-test. A lasting implementation of assessment will allow

for enhancements of the the program greatly.

EVALUATION PLAN FOR LLC 20

References

Creswell, J. (2009). Research Design: Qualitative, quantitative, and mixed methods approaches.

(3rd ed). Thousand Oaks, CA: SAGE Publications.

EVALUATION PLAN FOR LLC 21

Evans, N. J. and Reason, R. D. (2001), Guiding Principles: A Review and Analysis of Student

Affairs Philosophical Statements. Education Publications. Paper 22.

Rogers, P. J. & Goodrick, D. (2010). Qualitative data analysis. In J. S. Wholey, H. P. Hatry, K.

E. Newcomer (Eds.), Handbook of practical program evaluation (3rd ed.). San Francisco,

CA: Jossey-Bass.

Schuh, J. H. (2009). Assessment methods for student affairs. San Franscisco, CA: Jossey-Bass.

Tinto, V. (1999) Taking retention seriously: Rethinking the first year of college.

NACADA Journal: Fall, Vol. 19, No. 2, pp. 5-9.

EVALUATION PLAN FOR LLC 22

Appendix A
Staff and Faculty involved in LLC 2016-2017

Marilyn Krough: Faculty Director of Learning Communities, Academic Advising

Shannon Howes: Director, Student Leadership Development

Sam Siner: Assistant Director for Academic Support & Learning Communities, Residence Life

Timothy Griffin: Resident Director for Mertz Hall, Residence Life

Giancarlo Garcia: Resident Assistant for Mertz Hall, Residence Life

Martina Sullivan-Konyn: Resident Assistant for Mertz Hall, Residence Life

EVALUATION PLAN FOR LLC 23

Appendix B
LLC Monthly Meeting Topics

Fall Semester:

Meeting 1: What is the Leadership Learning for Social Change Learning Community?

Meeting 2: Exploring What Leadership Means to Us (Values and Congruence)

Meeting 3: Who Am I as a Leader? (Consciousness of Self)

Meeting 4: Working Together (Collaboration and Common Purpose)

Meeting 5: Working Together Within Communities (Citizenship)

Spring Semester:

Meeting 6: Revisiting Who We Are (Consciousness of Self)

Meeting 7: Privilege, Oppression, and Systemic Social Change (Controversy with Civility)

Meeting 8: Service Project (Citizenship and Change)

Meeting 9: End of the Year Banquet Celebration

EVALUATION PLAN FOR LLC 24

Appendix C
Required Academic Courses for LLC
EVALUATION PLAN FOR LLC 25
EVALUATION PLAN FOR LLC 26

Appendix D
Logic Model
EVALUATION PLAN FOR LLC 27

Appendix E
Invitation to Participate in Survey

Invitation for pre-survey:

Hello [Insert Student Name],

As a member of the Leadership for Social Change Learning Community, we would like to gain
understanding of your current leadership skills through a 5-10 minute survey. Completing this
survey will help the learning community work towards improving for the future.

Please follow the link below to the pre-survey. All responses will be kept confidential, to be used
only to improve the Leadership for Social Change Learning Community.

Thank you in advance for your time and thoughtful feedback

Invitation for post-survey:

Hello [Insert Student Name],

As a member of the Leadership for Social Change Learning Community, we would like to gain
understanding of of leadership skills you learned over this academic year through a 5-10 minute
survey. Completing this survey will help the learning community work towards improving for
the future.

Please follow the link below to the post survey. All responses will be kept confidential, used only
to improve the Leadership for Social Change Learning Community.

Thank you in advance for your time and thoughtful feedback!

Follow-up Post-Survey Message:

Greetings [Insert Student Name],

Recently, you were invited to participate in a post-survey to support the growth of the Leadership
for Social Change Learning Communitys future.

Please follow the link here to take the post-survey. This survey will take about 5-10 minutes of
your time to complete. Thank you for dedicating your time and helping us improve the
Leadership for Social Change Learning Community.

Thank you in advance for your time!

EVALUATION PLAN FOR LLC 28

Appendix F
Pre-Test Assessment
This survey is used to gain understanding of your current leadership skills. Please answer the
following questions below.

On the scale below, with 1 being not at all and 5 being very, share how confident you are
using the leadership skills you learned when engaging with different groups on Loyola
University Chicagos campus.
1:Not at all 2 3 4 5: Very

How empowered do you

feel engaging with staff
and faculty on Loyolas
campus?

How empowered do you

feel engaging with other
students outside of the
LCC on Loyolas
campus?

How empowered do you

feel engaging with
students in the LLC on
Loyolas campus?

On the scale below, with 1 being strongly disagree and 5 being strongly agree, share your
understanding of social identity in relation to the different listed concepts.
1:Strongly 2 3 4 5: Strongly
Disagree Agree

I can connect how social

identity is related to
leadership.

I understand how social

identity is related to
power and privilege.

I understand how social

identity plays a role in
social change.
EVALUATION PLAN FOR LLC 29

On the scale below, with 1 being strongly disagree and 5 being strongly agree, share your
comfortability in engaging in different types of advocacy.
1:Strongly 2 3 4 5: Strongly
Disagree Agree

I am engaging in
advocacy activities.

I am comfortable
engaging in
conversations about
advocacy.

What is your current gender identity?

Male

Female

Genderqueer

Gender Non-Conforming

Other (please specify)

__________________

What is your race? Please check all that apply.

White

Black or African American

Hispanic

Asian

Native American

Other (please specify)

__________________
EVALUATION PLAN FOR LLC 30

Appendix G
Post-Test Assessment
This survey will be used to gain understanding of the leadership skills you learned over this
academic year. Please answer the following questions below.

On the scale below, with 1 being not at all and 5 being very, share how empowered you feel
engaging with different groups on Loyola University Chicagos campus as a result of
participating in the LLC.
1:Not at all 2 3 4 5: Very

How empowered do you

feel engaging with staff
and faculty on Loyolas
campus?

How empowered do you

feel engaging with other
students outside of the
LCC on Loyolas
campus?

How empowered do you

feel engaging with
students in the LLC on
Loyolas campus?

On the scale below, with 1 being strongly disagree and 5 being strongly agree, share your
understanding of social identity in relation to the different listed concepts after being a part of the
LLC.
1:Strongly 2 3 4 5: Strongly
Disagree Agree

I understand how social

identity is related to
leadership.

I understand how social

identity is related to
power and privilege.

I understand how social

identity plays a role in
social change.
EVALUATION PLAN FOR LLC 31

On the scale below, with 1 being strongly disagree and 5 being strongly agree, share your
comfortability in engaging in different types of advocacy after being a part of the LLC.
1:Strongly 2 3 4 5: Strongly
Disagree Agree

I am comfortable
engaging advocacy
activities.

I am comfortable
engaging in
conversations about
advocacy.

What is your current gender identity?

Male

Female

Genderqueer

Gender Non-Conforming

Other (please specify)

__________________

What is your race? Please check all that apply.

White

Black or African American

Hispanic

Asian

Native American

Other (please specify)

__________________
EVALUATION PLAN FOR LLC 32

Appendix H
Invitations to Participate in Focus Group

Invitation for focus group:

Hello [Insert Student Name],

As a member of the Leadership for Social Change Learning Community, we would like feedback
of your experiences this past year. This feedback would be given through the form of a focus
group- along with a few of your fellow learning community cohort.

Within the bottom of this email is a e-form to indicate interest in participating in the focus group.
All material disclosed in the focus group will be kept confidential, used only to improve the
Leadership for Social Change Learning Community.

Thank you in advance for your time!

Follow-up Focus Group Message:

Greetings [Insert Student Name],

Recently, you were invited to participate in a focus group here to support the growth of the
Leadership for Social Change Learning Communitys future.

Please follow the link here to sign up for a focus group time. The focus group will meet once for
60-90 minutes. Thank you for dedicating your time and helping us improve the Leadership for
Social Change Learning Community.

Thank you in advance for your time!

EVALUATION PLAN FOR LLC 33

Appendix I
Consent to Participate in Focus Group

Project Title: Leadership for Social Change Learning Community Evaluation

Researchers: Emma Hindman, Candice Germany, and Dana Gramuglia

Moderator: Emma Hindman

Assistant Moderator: Dana Gramuglia

Introduction: Due to your participation in the Leadership for Social Change Learning
Community at Loyola University Chicago, you are being asked to participate in a focus group for
evaluation purposes.

Purpose: The main purpose of this focus group is to improve the quality of the Leadership for
Social Change Learning Community by exploring more in depth discussion than the pre and post
survey can provide. This will improve the experience of the future classes of the learning
community.

Procedure: The focus group will consist of seven open ended questions and space for additional
comments. This will last approximately 60-90 minutes of your time.

Risks/Benefits: With participation in the focus group there are no foreseeable risks involved.
There will be in depth reflections on the experiences within the Leadership for Social Change
Learning Community; and there is potential for personal risk depending on each persons
experiences. Benefits include learning of others experiences within the LLC and elevating the
experience of future classes in the LLC by providing data for assessment and implementation to
improve the program.

Compensation: There will be no monetary compensation for the participation in this focus group.
During the time of the focus group there will be refreshments provided.

Confidentiality: All information gathered from the focus group will be kept confidential. There
have been protocols and procedures put in place to ensure the identity of the focus group
participants. The answers, thoughts, and identity of those involved in the focus group will not be
revealed to anyone not listed as a moderator or assistant moderator.

Participation: The focus group is voluntary and can be left at anytime without consequences if
atmosphere of the space does not feel safe or becomes uncomfortable.
EVALUATION PLAN FOR LLC 34

Contacts and Questions: If there are any questions about the focus group, please contact the
moderator at ehindman@luc.edu and the assistant moderator at dgramuglia@luc.edu.

Statement of Consent: By signing below this indicates you have read the content above, asked
any outlying questions, and confirm the participation in the focus group. For your own record
keeping, we will provide a copy of this signed document.

__________________________________ ____________
Participants Signature Date

__________________________________ ____________
Moderators Signature Date
 EVALUATION PLAN FOR LLC 35

Appendix J
Focus Group Questions

Introduction:
Thank you for taking the time to attend todays focus group. My name is ______________, and
I am the moderator for todays discussion. The purpose of this focus group is to gain a better
understanding of your experiences within the Leadership for Social Change Learning
Community this year. You previously completed an electronic survey through LUCentral, and
this is an opportunity to share specific experiences and feedback you have. Your participation
and honesty in sharing is appreciated, and your responses will be used in an effort to continue to
improve the learning community in future years.

During our time together I will ask a series of different questions related to the learning
community. These questions will serve as a guide to our conversation. Feel free to participate as
you feel comfortable. This is meant to be informal, so there is no need to raise your hand or ask
permission to speak. We want to hear all of your different perspectives and experiences, and
encourage everyone to share freely.

Our conversation will be recorded and then transcribed. All of your names and information will
remain confidential in the final report. Are there any questions at this time? Please read over the
informed consent form that you have received, read over it, and sign your name at the bottom of
the page. When you have completed this, please place your pen down so that I can collect them.
Thank you. Before we begin, are there any other questions? I will now turn on the recorder.

Discussion Questions:
What made you want to apply to be a part of the LLC?
What friendships and connections came out of your experience in the LLC?
Can you tell me about the relationships you had with staff and faculty involved with the LLC?
Can you provide an example of a time you have used new knowledge and skills from the LLC?
Could you tell me about how you hope to enact positive social change after being a part of the
LLC?
What programs or experiences should we offer again for next years LLC?
What can be done to improve the LLC program in the future?

Closing
As we wrap up, I want to make sure everyone has had the opportunity to share what they wanted.
Is there anything that was not discussed that you would like to share at this time? Thank you. I
wanted to take the chance to summarize some of the conversations we had today, [moderator will
share themes and patterns that were shared throughout the focus group]. Does all of that sound
correct?
EVALUATION PLAN FOR LLC 36

On behalf of Student Leadership Development, Residence Life, and Academic Advising, thank
you for participating in todays focus group. Your experiences and feedback are appreciated and
valuable as we evaluate the Leadership for Social Change Learning Community and work
towards improving experiences for future students. I will be available for a few minutes
following our time together if you have any individual questions or thoughts you would like to
share.
EVALUATION PLAN FOR LLC 37

Appendix K
Budget

Activity Item Cost Per Item Quantity Total $

Survey Survey $0 $0
Administration

Focus Group

Audio Recording $0 1 $0
Equipment Rent from LUC Digital
Media Services

OrgSync $0
Provided through the
departmental budget

Refreshments (2 $2 8 $16
Liters)

Cups $2.00 2 $4.00

Consent Forms $0 42 $0

Room $0 4 $0
Reservations

Coding System SPSS $0 1 $0

Provided through the
departmental budget

Total Cost $20.00