The Evaluation of Introduction to Middle School Journalism Shanda Veatch Dept of Ed Tech, 512 Boise State University

Evaluation Criteria Effectiveness: Mastery of WBI goals

Explanation Determine if students are able to produce a school newspaper

Data Sources Student has contributed an article to the school publication Student has correctly formatted a page for the newspaper Practice exercises in WBI Student reflection on engagement level of graphics, video and text Instructor reflection about delivery system Alignment with Essential Academic Learning Standards Document instructor prep and delivery time Document student WBI activity time

Appeal: Gain and maintain learner attention and usability Review graphics, video and text for interest Review content for interest Review technology for navigability

Efficiency: Instruction delivered in a timely manner

Investigate time required for curriculum delivery

Evaluation Criteria and Categories Effectiveness Goals

Questions

Is the information accurate? Are the goals and objectives clear? Are goals and objectives achievable? Are the goals and content appropriate for the method of

Content

Technology

Message Design

delivery Is the information complete, covering the content properly? Is there a match among content, objectives, activities, and assessment tools? Are reference citations provided? Do the instructional activities reflect current practices in Journalism? Do the activities promote learning? Do the technology applications function properly? Were materials easy to access by students? Are copyright and intellectual property not violated? Do supporting graphics and features enhance the learning without distracting? Is appropriate voice used in expressing the content to the learners? Was humor used appropriately? Are directions clear? Was the time frame of the course appropriate?

Appeal Goals Content Technology

Are goals relevant to learners? Is the content interesting and challenging? Are typographical, spelling and grammatical errors distracting? Are there coding errors? Is the code written in a user-friendly protocol? Is navigation easy? Is the media pleasing? Is the vocabulary level and tone appropriate for the content and middle school learners? Are the screens uncluttered and with plenty of white space?

Message Design

Is the color, typeface, emphasis used appropriately and to enhance learning? Do supporting graphics and features enhance learning without distracting? Do the graphic devices function properly? Are the graphics, animations, or sound clear? Does it have good navigational design? Is the meaning of the icons clear and easy to use? Are the screen layouts appropriate middle school Journalism goals?

Efficiency Goals

Content

Technology

Message Design

Are the goals stated clearly and concisely? Is the purpose stated clearly and concisely? Is there congruence between the instructional goals and content? Is the content information clearly and concisely presented? Is it appropriate to middle school Journalism? Is it timely, up to date? Is access to the instructor or other learners provided? Is the website structured appropriately? Do technology applications function easily and efficiently? Is the organization and structure of the message coherent? Are the titles and subtitles used to organize the content? Is the time frame for the curriculum appropriate for middle school?

Introduction to Middle School Journalism Stakeholders Primary Stakeholders

Instructor: Shanda Veatch. She is responsible for delivery of instruction and coordination for the publication of the school newspaper. Designer: Shanda Veatch. Not only is she responsible for instructional delivery, but she is in charge of designing Introduction to Middle School Journalism and developing the evaluation plans. Students: The students at Gaiser Middle School will directly impact the WBI through the piloting process.

Secondary Stakeholders Michael Lane. He is the principal of Gaiser Middle School, and Shanda Veatchs direct supervisor. He will be providing feedback to Shanda about the quality of the publication and the program. Rose Crismon. She is the IT specialist at Gaiser Middle School. She will be working with Shanda Veatch to ensure student access to all elements and modules of the WBI. Gaiser Middle School community. This is a project based course, and the final product is presented to the students, staff and extended GMS community. The quality of this publication will represent Gaiser to this community. Materials to be Examined Objectives Assessment material Curriculum organization Delivery of instruction Student engagement level Navigation Interface with VPS server Graphics WBI prototypes

Design Plans

Prototype and Website

Evaluators and Reviewers Evaluator/Designer/Instructor: Shanda Veatch is responsible for creating the WBI and delivering the curriculum to students at Gaiser Middle School. She has taken many courses in Education Technology, and created multiple assessments, both formative and summative. She will be analyzing the data created curing the pilot of this curriculum for dissemination by her instructor at Boise State University and her supervisor at Gaiser Middle School

Expert Reviewer (Instructional Design): K Dianne Hall, Ed.D. Instructional Technology, Texas Tech University. Professor Hall is an adjunct professor at Boise State University and, as of the writing of this design document, is the instructor of Online Course Design for the Masters in Education Technology program at Boise State. End User Reviewer: Students at Gaiser Middle School who are participating in the Journalism program during the 2010-2011 school year will participate in the piloting of this curriculum. They will be providing feedback through formative evaluations as well as providing regular feedback during the piloting process. Expert Reviewer (Subject Matter): Rose Crismon is currently heading information technology department at Gaiser Middle School. She is certified CCNA and provides staff at both Gaiser and a local elementary school with all technology support. She is also network administrator for both schools.

Evaluation Criteria and Categories Effectiveness Goals

Question

Methods and Tools

Content

Is the information accurate? Are the goals and objectives clear? Are goals and objectives achievable? Are the goals and content appropriate for the method of delivery Is the information complete, covering the content properly? Is there a match among content, objectives, activities, and assessment tools? Are reference citations provided? Do the instructional activities reflect current practices in Journalism?

Expert, Instructor with checklists, End user review and survey, Extant data (practice exercises, final publication)

Expert, Instructor with checklists, End user review and survey, Extant data (practice exercises, final publication)

 Technology Message Design

Do the activities promote learning? Do the technology applications function properly? Were materials easy to access by students? Are copyright and intellectual property not violated? Do supporting graphics and features enhance the learning without distracting? Is appropriate voice used in expressing the content to the learners? Was humor used appropriately? Are directions clear? Was the time frame of the course appropriate?

Expert (ID, SME, instructor with checklists), End-user review/interview, and observation

Expert (ID, instructor with checklists), Enduser review, observation

Appeal Goals Content Technology

Are goals relevant to learners?

Expert (ID, SME), Enduser review

Is the content interesting Expert (instructor with and challenging? checklists), Observation Are typographical, spelling and grammatical errors distracting? Are there coding errors? Is the code written in a user-friendly protocol? Is navigation easy? Is the media pleasing? Is the vocabulary level and tone appropriate for the content and middle school learners? Are the screens Expert (ID, Instructor with checklists), Enduser observation, Enduser formative checklists

Message Design

Expert (ID, SME), Enduser survey, observation

uncluttered and with plenty of white space? Is the color, typeface, emphasis used appropriately and to enhance learning? Do supporting graphics and features enhance learning without distracting? Do the graphic devices function properly? Are the graphics, animations, or sound clear? Does it have good navigational design? Is the meaning of the icons clear and easy to use? Are the screen layouts appropriate middle school Journalism goals?

Efficiency Goals

Content

Technology

Are the goals stated clearly and concisely? Is the purpose stated clearly and concisely? Is there congruence between the instructional goals and content? Is the content information clearly and concisely presented? Is it appropriate to middle school Journalism? Is it timely, up to date? Is access to the instructor or other learners provided? Is the website structured appropriately?

Expert (SME, ID, Instructor with checklists), End-user observation and interview

Expert (ID, Instructor), End-user observation and interview

Expert (ID, SME and instructor with checklists), End-user observation, Extant data

 Message Design

Do technology applications function easily and efficiently? Is the organization and structure of the message coherent? Are the titles and subtitles used effectively Is the time frame for the curriculum appropriate for middle school?

Expert (SME, ID, Instructor with checklists), End-user observation

Introduction to Middle School Journalism Evaluation Methods and Schedule This program will be field-tested and formative assessments will be conducted throughout the design, implementation and review faze of this project. Review will be ongoing by the subject matter expert, the instructional designer, the instructor as well as the students who are testing this curriculum. Adjustments will be made as a result of their feedback at each stage of the process. There will be between 27-30 students participating in this program, and, as they are currently taking middle school Journalism at Gaiser Middle School, their participation will be compulsory. Permission for participation will not be necessary, as the individual participants will not be named. Modules will be delivered as they are created, in order to be modified to meet the needs of the learners. Delivery of the curriculum will not be in chronological order in this field test, but each module will be examined by the above listed experts as well as the end-users for logical flow from one to the next. Due to time constraints, the field testing timeline will not encompass an entire trimester. Depending on the feedback, this may have a substantial impact the final project length. Ongoing Decisions Made while Developing and revising the WBI Design and Prototype With each step of the implementation of the Introduction to Middle School Journalism field test, data will be collected from the end-users and curriculum will be modified to reflect the changes that need to be made as a result. Shanda will share all reported technical data incongruencies with the SME, Rose Crismon, in order to evaluate the best recourse for reconciliation of the data with the needs of the learners.

Students will be producing a full publication as a result of this WBI. The final project will be analyzed by the ID and the SME as well as the administrator at Gaiser Middle School, Michael Lane, and reviewed for attainment of learning goals and curriculum alignment with state standards. Inferential data, as it relates to end-user satisfaction will be shared with the team. The team will determine how that will impact full implementation the final product.

Preliminary Planning for Summative Evaluation Evaluation Criteria Main Questions Data Sources Effectiveness Did the final project Individual student meet with the activities instructional goals Goal review by and objectives? instructional , Did the WBI content administrative and and assessment align expert team with state standards? Efficiency What was the average Documented length of time classroom time per students were module engaged in each Documented whole module of the course? class instructional What was the total time time invested in the classroom for the WBI project as a whole? Appeal Will this program Delivery of WBI to stand alone without offsite instructor for instructional implementation in a assistance by the ID? separate classroom and review Can this WBI be replicated in other Review by instructors classrooms or as an in virtual learning entirely virtual environment presentation?

The time frame for this project extends beyond the trial period presented so far in this design document. The field testing will take place during the 2010-2011 school year. In order to meet the needs of the Instructional Designer, Shanda Veatch, who is designing this curriculum for her Masters at Boise State University, initial design must be completed by

December of 2010. A preliminary summative evaluation will not be complete until the end of the school year. Since there has never been a WBI for Journalism at Gaiser Middle School, there is no baseline data from which to draw any comparative data. There have been other curriculum delivery methods, and the comparative data will be based on the overall success of student scores as a whole drawn from the past three years. We will be looking for improvement in overall classroom grade point averages as well as decreased failure rates and increased student engagement among Journalism students at Gaiser Middle School. Shanda Veatch, as the instructor, will oversee the field testing of this curriculum, and as a result will utilize observation, questionnaires, surveys and interviews along with the assessment data to determine the effectiveness of this WBI. Formative evaluation data will be analyzed in order to improve the effectiveness of the delivery systems as well as the content. The summative evaluation will be provided in a preliminary report to the Gaiser Middle School administrative team at the end of the school year. The process will be undertaken again the following fall of 2011, allowing for a second summative evaluation to be performed before the end of that year. This evaluation will be provided to the stakeholders at the end of that school year as a formal report, whereupon the curriculum will be delivered to another teacher for instruction during the 2012-2013 school year. At the conclusion of this third year, the findings of the new instructor will be provided to the Instructional Designer, and a final summative evaluation report will be drafted for delivery to the head of the Careers in Technical Education at Vancouver Public Schools. It is with hope that this curriculum will be available for regular delivery by middle school teachers in the Vancouver School District at that point. Pre-Planning Activities WBI Project Timelines Tasks Conduct preplanning activities Persons Responsible Veatch Projected Week 1 Week 2 Week 3 Week 4 Week Week Week 5 6 7

Actual

Write Objectives

Veatch

Projected

Actual Write assessment items and tools Evaluate objectives and assessment items Cluster and sequence objectives Veatch Projected

Actual Veatch Projected

Actual Veatch Projected

Actual Create WBI strategy worksheet Veatch Projected

Actual Identify media Veatch Projected

Actual Evaluate Veatch instructional strategies and media selections Flowchart and Veatch Projected

Actual Projected

storyboard lesson

Actual

Week Week Week Week Week 8 9 10 11 12 Flowchart and storyboard lesson (Cont) Design website Veatch Projected

Actual Veatch, Crismon Projected

Actual Convert storyboard to web pages Veatch Projected

Actual Projected

Conduct Veatch, evaluation Crismon (test website and instructional quality of WBI, troubleshoot technology, test Final Project Due Veatch

Actual

Projected

Actual

Learning Task Item and Number Students will create a newspaper for publication

Task-Objective-assessment Item Blueprint For Introduction to Middle School Journalism Objective Learning Outcome Given the articles and accompanying photographs and graphics, students will use InDesign to produce a newspaper for school-wide distribution Intellectual skill: Synthesis, generalization Is the paper free of grammatical errors? Are the pictures clear? Does each picture have a caption? Is there a headline and byline for each article? Are the articles formatted correctly? Is there a border on every page?

1.0 Students will lay out a page for the newspaper

Students will use InDesign to lay out a page for the newspaper

Intellectual skill: Synthesis, retrieval

Is the page formatted correctly? Does each article have a Headline and byline? Does each article have a graphic or

photograph? Does each photograph have a caption? 1.1 Students will select the articles and pictures for publication Students will read and analyze the articles submitted for the publication and choose the ones for publication Intellectual skill: Evaluation, reinforcement Which articles are formatted correctly? Which articles are free of grammatical errors? Which pictures are clear? Which articles are most relevant to the current issue? 2.0 Students will write a human interest story Having interviewed a subject, students will write a 200-250 word article including direct and indirect quotations, and photographs of the subject Intellectual skill: Synthesis, responding Is the article a narrative, with quotations woven into the text? Are the pictures clear and easy to view? Intellectual skill: Application, responding Is the subject centered in the frame? Is the lighting conducive to publication?

2.1 Students will photograph their subjects

Students will take several photographs of their subjects during the interview

Is the photo clear and easy to view? 2.2 Students will compile a list of quotes from their interview Students will record at least 10 quotes from the interview Intellectual skill: Analysis, retrieval Which quotations define the personality? Which quotations provide insight into the subject? Which quotations help to further the story? 2.3 Students will conduct an interview Students will schedule and arrange an interview place and time Intellectual skill: Application, responding Did the student contact the interviewee in advance of the interview to schedule a time? Was the student prepared with questions, camera and writing tools for conducting the interview? Did the student check the quotes with the

interviewee before dismissing his or herself? Did the interviewer thank the interviewee? 2.4 Students will prepare a list of questions for the interviewee Students will generate a list of at least 20 questions that they can ask the interviewee Intellectual skill: Synthesis, retrieval What was the reason for choosing this subject? Are the questions relevant to telling a story? Do the questions elicit extended responses from the interviewee? 2.5 Students will choose a subject to interview Students will choose an interviewee and explain their rational for choosing this person Intellectual skill: Analysis, selective perception What makes this person relevant to school culture? What would you like to learn about this person? 2.6 Students will identify appropriate interviewing etiquette Students will create an interviewing etiquette themed powerpoint and conduct a mock interview with a classmate, Intellectual skill: Application, synthesis, semantic encoding How do you set up an interview? How do you

demonstrating correct interviewing etiquette

approach an interviewee when beginning the meeting? How do you ask for clarification? How to ask follow up questions? How to check for inaccuracies in the quotations? How to close the interview

3.0 Students will write an editorial or draw an editorial cartoon

Students will choose from the editorial topics to write an opinion piece on this subject

Intellectual skill: Synthesis, responding

What about this subject do I agree with or disagree with? What is the counter argument? What is my solution to the problem?

3.1 Students will list appropriate editorial topics for the newspaper

Students will discuss relevant school-related topics, and choose at least three for the publication

Intellectual skill: Evaluation, retrieval

What are some school-related debates that are happening on campus and in the

classrooms? 3.2 Students will identify the structure used in opinion writing Students will be able to identify the structure of opinion writing (thesis statement, argument, counter argument, solution, restatement of the thesis) Intellectual skill: Knowledge, responding How do I organize my writing for an opinion article? What is the difference between a news article and an opinion article? Why would my audience find this subject interesting? What can I do to make them want to read on? 4.1 Students will select feature topics for the newspaper Students will brainstorm a list of school-related topics to write about Intellectual skill: Evaluation, responding What are some successes that have been noted on campus? What are some changes we have witnessed? What makes us unique as a community? 4.2 Students will When presented with two Intellectual skill: Based on my

3.3 Students will learn the characteristics of opinion writing

Given two pieces of writing, Intellectual skill: one a news story and one an Application, editorial, students will be selective perception able to discern which is an editorial with 100% accuracy Students will organize their ideas into a short 200-250 word feature article on one of the selected topics Intellectual skill: Synthesis, generalization

4.0 Students will write a feature article for the newspaper

identify the differences between news writing and feature writing

articles, one a news article and one a feature story, students will identify the feature story with 100% accuracy

Application, semantic encoding

prior experience, what tools do I now have to help me understand the differences between these two styles of writing? How is a feature story organized? What makes it unique to journalistic writing?

4.3 Students will identify the characteristics of feature writing

Students will be able to explain the essential elements of feature writing (narrative/expository, catchy opening, sequences information, reaches a conclusion) with 100% accuracy Given the topics generated, students will write a short news story employing the 5Ws and the H

Intellectual skill: Knowledge, semantic encoding

5.0 Students will write a news story for the newspaper

Intellectual skill: Synthesis, generalization

Did I identify the who, what, where and when at the beginning of the article? Did I discuss the why and the how at the conclusion of the article?

5.1 Students will choose newsworthy topics for the newspaper

Students will generate a list of at least 4 school-related news stories for publication in the paper

Intellectual skill: Evaluation, retrieval

What are some things that have occurred on campus that may not be widely known?

What athletic events, social events, classroom events or extracurricular events have been reported to you lately? 5.2 Students will identify the features of the inverted pyramid structure of a news story 5.3 Students will identify the essential elements of a news story Students will identify the 4 Ws as the pyramid base and the Why and How as the tip on a graphic organizer Intellectual skill: Application, selective perception What is the inverted pyramid and how will it help me to write a story? What happened to whom? Where did happen? When did it happen? Why did it happen and how did it happen? 6.0 Students will determine the format of the school newspaper Student will determine the Intellectual skill: order of sections for the Evaluation, school newspaper (ie: news, generalization feature, human interest, sports, editorial/opinion, games/puzzles/art/poetry) What makes a newspaper readable? How will our paper entice readers to read on? What sections can we find in the local

Students will Students will be able to answer the 5 Ws and the H after reading a short news story

Intellectual skill: Application, selective perception

6.1 Students will identify different sections in a

Students will create a virtual collage using InDesign in order to identify the different

Intellectual skill: Application, synthesis, semantic

newspaper

sections of a newspaper (news, sports, feature, opinion, lifestyle)

encoding

paper? What sections can we find in our school newspaper?

6.2 Students will identify vocabulary related to newspapers

Students will identify the parts of a newspaper with at 100% mastery. (Masthead, headline, byline, caption, cutline)

Intellectual skill: Knowledge, semantic encoding

What are the parts of the newspaper? Where will I find the masthead, headline, byline, caption and cutline in a traditional newspaper?

6.3 Students will identify the purposes of a newspaper

As a class, students will create a list of things that can be found in a newspaper

Intellectual skill: Application, reception

Why do you read the newspaper? Why do your parents read the paper? What other reasons can you

Introduction to Middle School Journalism Objective Clusters 1. 2. 3. 4. 5. 6. Create your page! (1.0, 1.1) Writing a profile (2.0- 2.6) Whats your opinion? (3.0- 3.3) The feature article (4.0- 4.3) Time for the news (5.0-5.3) Whats a newspaper? (6.0-6.3)