Purpose of Evaluation

This evaluation was conducted to determine if the instructional goal of the Mixdesign billing process was met. The evaluation should aid in determining if the lesson plan, materials, presentation, and workshop as a whole is in need of revisions, additions, or improvement in order to better meet the instructional goal. The instructional objectives were reviewed and those who participated were provided the pre-assessment methods as outlined in the lesson plan. The #oal
$The staff of mixdesign will better understand the billing process and therefore properl! track their time on assigned projects.%

udience
The recipients of the evaluation is majorl! m!self, however, the results will be shared with upper management in order to assess whether the materials will be implemented for an actual run of the workshop.

"ssues
ll materials were evaluated against the objectives designed for the workshop.

Formative & Summative Evaluation

&esources
The data was collected in the conference room of mixdesign. The 'ME and target individuals re(uired materials listed below) *omputer "nternet *onnection +e!board Printed ,andouts Pen-Pencil

.ata
The data was collected in the mixdesign conference room. n online surve! for both formative and summative evaluation was administered via 'urve! Monke!.

Evidence
Formative & Summative Evaluation

/ormative Evaluation) 'ME and additional individual viewed the introduction module with the use of the computer and its mouse the participants were given the introduction to the module and were administered the overview post assessment and formative evaluation surve! 0 ppendix 12. 'ummative Evaluation) 'ME and additional individual were administered the workshop as it was intended to be followed. Participants were given the same instruction b! beginning with the Module 3verview and provided with the post assessment. The workshop continued with the .igital /older ,andout and real time execution with the facilitator. The participants did not receive three hours to track time on their project, however the! were explained how the workshop will be completed in real-time. /ollowing the workshop the participants were given the final essa! assessment 0 ppendix *2 and an online surve! 0 ppendix .2.

Methodolog!
.ata gathering Techni(ues /ormative Evaluation included the introduction overview post assessment data and the /ormative 'urve! Evaluation data was gathered via 'urve! Monke!. The 'ummative Evaluation includes an essa! covering the process and each person4s role during the process.

/ormative Trials

'ME) 1illing .epartment Observational 5 'ME maintained excited expression throught introduction module. 6erbal comments included $so true, this is accurate, and this is a reall! cool wa! to explain it.% Post Assessment The 'ME correctl! answered all (uestions. Total time took 7 minutes8 this was three minutes faster than the *reative .esigner. 9uestion number there4s answer was ver! specific and perfectl! written in relation to workshop objectives) How is billing related to the Project Process? Billing is directly related to the project process. Without proper tracking of jobs, codes, and employee time spent on each process, the billing department cannot properly bill clients in the most efficient and timely manner. Additionally, failing to properly follow the process and document time correctly can cause companies to lose money, as well
Formative & Summative Evaluation

as inaccurately invoice clients. Survey :. The materials directl! relate to the learning objectives; 5 Strongly Agree <. .o !ou feel !ou were given sufficient instruction during the workshop introduction module; 5 Yes =. .o !ou feel !ou met the objectives of the introduction module of the workshop; 5 Yes >. ,ow practical are the workshop materials as the! relate to repeat use and instruction; 5 Vey Practical 7. ,ow would !ou (ualif! the workshop materials; 5 E ceeds E !ectation ?. ,ow would !ou (ualif! the workshop instruction; 5 E ceeds E !ectation @. ,ow would !ou (ualif! the workshop organiAation; 5 E ceeds E !ectation B. ,ow would !ou (ualif! the clarit! of the workshop; 5 E ceeds E !ectation C. .o !ou feel the billing process at mixdesign is logicall! covered; 5 Yes :D. Please provide an! feedback that relates to the materials and instruction so The process is very accurate to real life situations at mixdesign. The Prezi was the most helpful tool as it took what in any other circumstance could be a dry presentation, and made it fun and usable for high level designers. Overall it is a good time for introduction to first time users in our process and for retraining others how to properly submit timesheet for tracking and billing.

Additional" 'r. *reative .esigner Observational 5 There were several facial expressions to suggest the designer appeared visuall! stimulated. Ehen time tracking was covered in the presentation, the designer seemed uncertain about how the real-time design would work. Post Assessment 5 The designer did not struggle with the assessment. The (uestion) ,ow is billing related to the Project Process was vague reflecting there ma! have been low comprehension.

Formative & Summative Evaluation

Survey ::. The materials directl! relate to the learning objectives; 5 Agree :<. .o !ou feel !ou were given sufficient instruction during the workshop introduction module; 5 Yes :=. .o !ou feel !ou met the objectives of the introduction module of the workshop; 5 Yes :>. ,ow practical are the workshop materials as the! relate to repeat use and instruction; 5 Practical :7. ,ow would !ou (ualif! the workshop materials; 5 Su##icient :?. ,ow would !ou (ualif! the workshop instruction; 5 Su##icient :@. ,ow would !ou (ualif! the workshop organiAation; 5 E ceeds Expectation :B. ,ow would !ou (ualif! the clarit! of the workshop; 5 Su##icient :C. .o !ou feel the billing process at mixdesign is logicall! covered; 5 Yes <D. Please provide an! feedback that relates to the materials and instruction so it ma! lead to workshop improvement and revision. ! liked that this was a one day workshop where we all get to go over everything as a team. "y favorite part was the presentation. The only thing ! think needed to be thought about more was that the time tracking is for the same pro#ect which doesn$t really consider multiple pro#ects. The detail needed for each hour needed to be stressed more.

'ummative Trials

'ME) 1illing .epartment Observational - 'ME appeared to be ver! engaged b! the materials and often shook head in approval at working and instruction. 3verall was excited and enthusiastic during trial. Post$Assessment 5 'ME clearl! and efficient articulated the billing process and each person4s role. 'ME was concise and accurate and clearl! tied back objectives to answer. Survey - 'urprisingl! the 'ME got the time increment (uestion wrong. Ee track time in D.<7 increments until the! e(ual an hour i.e. :.<7. The! answered :. " know see how this needs to be clarified. 'ME also did not properl! place stages in correct order. fter completion stated the! misread and after submission had realiAed their error. This comment was not solicited, was stated without provocation. :. "n what increments of time should mixdesign emplo!ees track and log their time; 5 % <. ,ow often should !ou submit !our timesheet; 5 &ee'ly =. Place the billing process steps in proper order. 5 ( incorrect >. The activities and materials met the goal of the workshop. 5 )rue
Formative & Summative Evaluation

7. The module overview presentation was relevant and informational. 5 Strongly Agree ?. The workshop and its instruction was interesting. 5 Strongly Agree @. .o !ou feel !ou can track time in accordance with the workshop materials; Yes B. .o the final assessment measure the understanding of the workshop objectives; - Yes C. .o !ou feel !ou better understand the billing process at mixdesign. Yes :D. Fse the space below to provide feedback based on the workshop and its process as a whole. Again, this accurately follows a logical way to track and submit time. This is often over looked by carelessness in employees, so it is nice to have everything drawn out and put into an official process. Additional" 'r. *reative .esigner Observational 5 .esigner appeared engaged through each section. .esigner commented on the design brief to be well-planned and filledout for emplo!ee use for the workshop. .esigner acted as appeared to find the digital folder portion simple and commented it was a simple step and ma! not be needed. The designer commented that the timesheet entr! portion of the workshop was useful as it aided in the understanding of the importance to each categor!. ,e stated the workshop, as a whole ma! need to stress each individual4s role. Post$Assessment 5 The designer was able to generall! explain how each person worked into the entire project process. ,e could have been a little more detailed in his explanation of the creative director. ,e was able to articulate the importance of proper time tracking.

Survey 5 :. "n what increments of time should mixdesign emplo!ees track and log their time; 5 *+(, <. ,ow often should !ou submit !our timesheet; 5 &ee'ly =. Place the billing process steps in proper order. - All -orrect >. The activities and materials met the goal of the workshop. 5 )rue 7. The module overview presentation was relevant and informational. 5
Formative & Summative Evaluation

Agree ?. The workshop and its instruction was interesting. 5 Agree @.. .o !ou feel !ou can track time in accordance with the workshop materials; 5 .nsure /t is hard to have to trac' time every hour+ /t will mean changing my wor' habits+ B. .o the final assessment measure the understanding of the workshop objectives; .nsure $ /t made me go over the ste!s o# the !rocess0 but / still thin' )he role o# each !erson needs to be stressed more+ C. .o !ou feel !ou better understand the billing process at mixdesign. Yes :D. Fse the space below to provide feedback based on the workshop and its process as a whole. I think that will a few small tweaks it will be really successful for the company.

&esults G

nal!sis

Evidence *ollection) Evidence was collected via 'urve! Monke! and essa! submission on the billing process. Evidence was also collected through paper submission of post assessments for both the formative and summative evaluations. Evidence nal!sis) 'urve! Monke! provides anal!tics on the surve!s. ,owever, " also evaluated the answers for the value placed on the multiple areas the surve! assessed. These answers also included a final feedback opportunit! so an! additional thoughts, revisions, or needs could be addressed in regards to the workshop, and its content. The same (ualitative anal!sis was conducted on the essa! assessment, which intended to be administered at the end of the workshop. This essa! helps to assess the full understanding of the billing process and each step that is involved in relation to the entire staff of mixdesign.

&eporting

Formative & Summative Evaluation

" found that as a whole the workshop was well liked and found to be a valuable instructional tool. fter anal!Aing feedback " realiAe the designers ma! not enjo! changing their regular work habit to track time more often. " believe if " ran a small group trial " would anticipate having the same results. There ma! be slight fluctuation in surve! answers. " learned that there ma! be some small tweaks needed, but that for the most part this workshop should be an effective instructional tool that can not onl! provide an engaging wa! to handle time-tracking, but also create a resource to grown billing efficienc!. " will use the above information to be sure to clearl! explain how the increments work so that ever!one understand we track time in increments of D.<7 up to : hour. /+E+ D.<7, D.7, D.@7, :, :.<7, :.7D, ::.@7, <.

/ormative Evaluation 'trateg!

" chose the 3bjective-1ased /ormative Evaluation as it directl! relates back to the founding objectives and the goal of the workshop. " find the best wa! to evaluate items is to tie them back to the reasons the! were created or designed. Ehile this t!pe of evaluation ma! not be best for ever! learning situation, it is a checks and balance for the materials to ensure the! are properl! related and designed for the appropriate intentions.

'ummative Evaluation Plan)

.etermining Program Effectiveness) " would primaril! use a surve! method for evaluation. portion of this evaluation will include an essa! test. ,owever, observation is a powerful evaluation method " intend to execute. " find, especiall! when technolog! is involved, observation leads to insight and thoughtful consideration of materials. " expect to be able to compile a great deal of data. The evaluation methods " have assembled will provide visual cues at areas that ma! need revisions. 3bservation will also lead to technical problems or workshop application feasibilit!. The surve! will provide application and retention data, but will also cover the direct feedback of participants. The essa! will also allow for the application of knowledge in an organiAed manner to which will express comprehension. " expect to find that participants will not have knowledge on the need for proper time tracking. " expect the! will not have realiAed that each member of the mixdesign team is a part of the billing process, which stems from the creation of a project and its job number. " expect the!
Formative & Summative Evaluation

will have CDH comprehension of the workshop and will be able to appl! it as intended.

.etermining Program Efficienc!)

The workshop is expected to be a full da! with an hour lunch break instead of the general half-hour lunch. Two instructors will be needed to facilitate the workshop that has been designed to be the 1illing .irector and the *reative .irector for the compan!. The entire staff of ten people will be re(uired to attend. The workshop will utiliAe the mixdesign conference room, which will accommodate all of those in attendance and will be outfitted with the proper technologies for execution.

.etermining *osts)
.evelopmental *osts) PreAi - ID.DD 3ffice Eord - I:DD.DD Emplo!ee Eorkshop .esign Time - I<7D.DD )otal" 12,*+** 3perational *osts) "n-house technolog! - ID.DD 3ne .a! of Jipsco - I<DD.DD *olored Printed Materials - I>D.DD )otal" 1(3*+** KJ"P'*3 cost determined b! billing department who is 'ME on operational costs for mixdesign office. "nstructional *osts) ,ourl! Emplo!ees) IBD.DD 'alar! Emplo!ees) IB7D.DD )otal" 142*+** 5rand )otal" 1%0,(*+** " believe the program will be cost-effective. Evidence supports that the instructional materials have been designed successfull!. Eith successful training the staff of mixdesign can properl! track their time that will lead to efficient billing. n improved and efficient billing department will lead to better cash flow for mixdesign. This in turn will be more cost-effective as it should lead to more than I:,7<D.DD in profit.

Formative & Summative Evaluation

Formative & Summative Evaluation