Grant Evaluation Report

 

Building A Model Technology-Rich Pre-service Teacher Education Program With An Emphasis on Selecting, Developing, and Using Online

Standards-Based 21st Century Content

 

(PR/Award No. P342A000048)

 

 

 

 

2000-2003

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Evaluation Report Prepared by

Dr. Bill Warner

The Bill Warner Group

 

June 2003

 

 


Grant Evaluation Report

June 2003

Project evaluation was provided by an external evaluator, Dr. Bill Warner, The Bill Warner Group. Dr. Warner was the evaluator for the Capacity Building grant in 1999. He has worked with the Implementation Grant since the start date, providing evaluation continuity throughout the life of the grant. Dr. Warner also was a member of the grant Technology Leadership Team (TLT) that met weekly to provide overall direction and on-going implementation strategies. The foundation for the evaluation of ÒBuilding a Model Technology-Rich Pre-service Teacher Education Program with an Emphasis on Selecting, Developing, and Using Online Standards-Based 21st Century ContentÓ grant was the logic model developed in the early stages of implementation. The logic model defines the background factors, project services, external services, grant outcomes and data sources. The logic model is found in Appendix A. The logic model was the foundation of the grant evaluation, showing the overall picture of the evaluation process. An Evaluation Worksheet (Appendix B) identifies the data sources used for each of the grant objectives. This worksheet was helpful in ensuring that each objective was addressed in the evaluation plan.

 

Each of the grant objectives will be reviewed in this report. The review will include a definition of success, evidence of the level of achievement or implementation, an analysis of the progress, and identification of the products produced as a result of grant activities. In summary, all grant objectives have been accomplished. The grant project allowed the Augsburg College Education Department to integrate technology into all courses, providing future teachers with technology skills that will allow them to be successful in the classrooms of tomorrowÕs schools.

 

 

 

 

 

 

 

Evaluation/Project Results

 

Objective 1

Weave technology into all education courses at Augsburg - technology will no longer be an "additional" course. (GPRA: 1.1, 1.2, 1.3, 1.4, 2.1)

Definition of Success

All Education Department faculty will integrate technology into the delivery of teacher training courses. Initially each faculty member will integrate the technology tasks defined by the ÒDigital Dozen.Ó (Appendix C). Faculty will utilize technology to present course content as well as require students to utilize technology in completing course requirements.

Progress Evidence

Accomplished.

All full-time faculty have integrated technology into the delivery of teacher education courses. All syllabi have been reviewed for technology integration. This review process performed by the grant evaluator documented the use of technology by faculty and requirements of students to use technology as part of the courses. Syllabi from the first year of the grant (2000) were compared to syllabi at the end of the grant. All Digital Dozen technology tasks have been integrated. This finding was cross documented through a Course Technology Survey that was administered to a majority of students in the education courses during the spring semester 2003. Survey results indicated that faculty were using technology to deliver courses and students were required to use technology in course assignments and assessments. A summary of the syllabi review and the student Course Technology Survey is in Appendix D and E. In addition, as a result of the survey analysis the education faculty decided not to discontinue the technology orientation course required of all students. Students reported that the course was valuable in understanding the uses of technology in education. The Technology Leadership Team (TLT) and faculty continue to have conversations regarding the integration of technology into education courses, centering on the twelve technology tasks that will be integrated into current courses with existing resources. Focus groups with faculty were held in the fall of 2001 and again in the spring of 2003 (Appendix F). The comments by faculty indicate that they are comfortable with technology integration and they have made significant progress in integrating technology into course requirements. The feedback surveys from the many grant related workshops and events indicated that they were diligent about gaining technology skills. The journal comments from faculty indicated that they were developing their technology skills and were integrating technology into their courses.

Analysis - Key Findings & Supporting Data

This objective was accomplished through a long term plan to provide faculty with the technology skills needed to integrate technology into their courses. The plan included group training sessions, special interest training sessions, technology conferences, off campus classes and one-on-one coaching. A full-time technology mentor provided Òjust-in-timeÓ technology support to faculty and students during the three years of the grant. The technology coach had an office in the education department, providing easy access to faculty. Stipend incentives were critical to the accomplishment of this objective.

However, few adjunct faculty were involved. Since adjunct faculty teach primary in the weekend college, it was difficult to schedule sessions that did not conflict with their teaching and other work schedules. Even though special efforts were made to involve adjuncts, results were minimal.

Products Produced

All Education Department syllabi have identified technology content and student requirements. All technology tasks identified in the Digital Dozen were integrated into course delivery and course requirements.

 

 

Objective 2

Have faculty serve as role models for prospective teachers, for each other, and for faculty throughout the College. Faculty will, in the eyes of cooperating K-12 schools, be well-versed in K-12 technology. (GPRA: 1.1, 1.2, 1.3, 1.4, 2.1, 3.2)

Definition of Success

All Education Department faculty will use a variety of technologies when teaching courses. Faculty will demonstrate how to effectively use technology support resources (on-line, mentors, help lines, other faculty and staff). Faculty will use technology when presenting to faculty at college-wide or departmental meetings. Faculty will work collaboratively with K-12 personnel to have preserve teachers integrate technology into teaching and learning.

Progress Evidence

Accomplished.

A foundation was developed established that enabled faculty to serve as role models in technology infusion. All faculty increased their technology expertise and are using technology in their teaching and office settings as evidenced from focus group discussions (Appendix F) and the student Course Technology Survey (Appendix E). Students reported that faculty were integrating technology into education department courses. Advocate focus groups (Appendix G) reported that faculty were giving more emphasis to technology. While all are not ready to be recognized as role models, significant progress has been made. Collaboration with partner schools proved a challenge and there was little opportunity for most faculty to provide support to K-12 teachers. Supervising teachers provided the best opportunity for faculty to serve as role models when working with student teachers.

Analysis - Key Findings & Supporting Data

Faculty are at various technology skill levels and need time to continue to learn, understand, and integrate technology into their courses. Gender differences were found among the faculty in the acceptance and ease with which technology is embraced! Of the 12 education department faculty members, the two male faculty members are well-versed in the use of technology, while most of the nine women faculty members have struggled to become tech-savvy. However, all made tremendous progress and are peer-tutoring each other.

 

Work with partner schools proved to be a difficult process. With the exception of collaboration with Eagan High School, the partner schools were not active participants in the grant. Eagan High School was a significant participant and allowed Augsburg faculty and opportunity to collaborate with secondary faculty.


 

Products Produced

All Education Department syllabi have identified technology content and student requirements. The project produced four technology monographs that have been made available to Augsburg faculty, students and K-12 partners.

 

 

Objective 3

Establish a cohort of prospective teachers who will work in partnership with faculty and in-service K-12 teachers to support the implementation of technology-based and content-rich instruction in K-12 classrooms. (GPRA: 2.1, 3.2)

Definition of Success

A cohort of current education students will be recruited each school year to participate in the Student Technology Advocates Program. Weekly meetings were scheduled to provide Technology Advocates with the needed skills to support the implementation of technology-based and content-rich instruction in their schools and classrooms, and to serve as technology assistants to college faculty. A work plan was developed for each Technology Advocate regarding knowledge, skill and leadership development activities.

Progress Evidence

Accomplished.

The cohort of prospective teachers, the Student Technology Advocates, was established and successfully operated since January 30, 2001. During the 2000-01 school year there were 20 students who participated as advocates. During the second year (2001-02) there are 18 student advocates. The third year (2002-03) generated 8 students who participated. The advocates met once a week each semester for two hours. The advocates formed an Advocates Leadership Team that recruited advocates, planned meetings and become the liaison link with the Grant Leadership Team. While fewer students participated each year, the students who participated became key technology supporters. As the participants moved into student teaching assignments they were able to demonstrate their technology skills to the students in their classes, their supervising teacher and the education faculty. The advocates reported in focus groups (Appendix G) that technology is a key component in being an effective teacher and they must constantly improve their technology skills. Advocates were provided access to e-Portfolios. All advocates were provided the opportunity to identify their technology skills by taking a Technology Skills Survey (Appendix H). The survey provided the advocates with feedback on their technology skills and an opportunity to develop their skills in the advocate meets.

 

The technology advocates represented a diverse range of students and skill levels: full-time and weekend college; high technology skills to low technology skills; elementary and secondary majors; males and females; and traditional age to non-traditional age. Incentives to participate included campus-wide recognition as technology advocates, opportunities to interact with local, state, and national tech-savvy leaders, attend technology conferences, and work on technology projects.

Analysis - Key Findings & Supporting Data

The Student Technology Advocates program made a significant contribution to the grant, in that education students became involved in technology. The Advocates Program attracted a small group each year, but provided an opportunity to work with students to develop their technology skills. Focus group results indicate that the Advocates have benefited greatly from the opportunity to become well-versed in the use of technology in their teaching, and have indeed recognized that they cannot be a millennium teacher without knowledge and skills in technology. One of the barriers was scheduling meetings, events, and activities at a time when a larger group of students could participate.

Products Produced

The Advocate Program produced a group of technology leaders at the college, who have applied their technology skills in education courses, student teaching and teaching positions. The Advocate Program developed technology leaders in the education department. The advocates produced e-portfolios, monograph articles and course projects.

 

 

Objective 4

Select, develop and use online technology- and standards-based content. (GPRA: 1.1, 1.2, 1.3, 1.4, 2.1)

Definition of Success

All Education Department faculty will integrate online technology and standards-based content into the delivery of teacher training courses. Each faculty member will integrate the technology tasked defined by the ÒDigital Dozen.Ó The integration of technology will include faculty using the collegesÕ online system, AugNet, and Blackboard, an online course management system.

Progress Evidence

Accomplished.

The full-time faculty have integrated technology into all the courses in the education department and all courses use the on-line format of Blackboard. There has been an excellent foundation established to use online technology and standards-based content. The use of online technology and standards-based content has grown significantly as a result of this sound foundation. The Augsburg faculty have been using a standards-based approach for many years. The education departmentÕs curriculum is designed to incorporate licensure standards and Minnesota graduation standards into the delivery of their courses. The Standards of Effective Practice for Beginning Teachers provide a framework for the teacher preparation program at Augsburg. Preservice teachers use the Minnesota Graduation Standards as a framework for planning lessons for K-12 students. All lessons written are aligned with these standards. The syllabi review documented standards for each course.

 

Faculty were introduced to the ISTE standards for technology and use the standards and others as guides for developing preservice teachersÕ technology skills. The faculty and technology advocates participated in the ISTE-based Technology Skills Survey - TAGLIT (Appendix H) for the second time this year. The results of this yearÕs survey were compared with the survey completed during the first year. With this comparative knowledge faculty and preservice teachers are better able to develop and monitor their own technology skill development. Some general conclusions from the comparison showed:

á       Both faculty and advocates had increased their technology skills.

á       There continues to be a wide variance in knowledge and skill levels within the faculty.

á       There continues to be a wide variance in knowledge and skill levels between students.

á       Faculty and students are increasing their breath and depth of technology skills.

á       Faculty and student opinions are similar regarding technology and the way the classroom works in the areas of cooperative learning, high level thinking, real world relevance, teacher as coach, and assessment.

 

As reported in Objective 1, the faculty identified a list of twelve technology tasks (Digital Dozen) that were integrated into current courses with existing resources. Many of these tasks provide an on-line feature to the presentation of teacher education courses. The student Course Technology Survey (Appendix E) results show that faculty are using technology to teach courses and require students to use technology to complete course requirements.

Analysis - Key Findings & Supporting Data

All full-time faculty have integrated technology and standards-based content into education courses. This is documented in by an analysis of education department syllabi for the start of the project to the most recent semester. In addition, students indicated in a survey that faculty are using technology to teach courses and they are required to use technology in course assignments. There was not as much success in achieving this objective with adjunct faculty teaching evenings and weekends. The grant was not able to connect to many of the adjunct faculty due to their work and teaching schedules.

Products Produced

All Education Department syllabi have identified technology content and student requirements. All technology tasks identified in the Digital Dozen were integrated into course delivery and course requirements.

 

 

Objective 5

Design, implement and demonstrate an online approach to sharing experiences and lessons learned in building a technology-based preservice teacher education program. (GPRA: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1)

Definition of Success

All Education Department faculty will provide periodic electronic updates related to the achievement of their assigned project goals. Faculty will use electronic portfolios to keep track of the project related activities/events and skill development. The Technology Advocates will use electronic portfolios to maintain a record of their skill development and examples of projects completed.

Progress Evidence

Accomplished.

The Augsburg education department website has a MT2 site that provides information about the Augsburg/TIES grant and the national PT3 projects. The address is: www .augsburg.edu/education/MT2.

 

Faculty provided periodic updates to the Technology Leadership Team (TLT) using e-mail. The updates highlighted the use of technology in their classes, use of technology by their students, and skill and knowledge upgrading activities. In addition, the TLT communicates with the Technology Advocates using e-mail. All faculty participated in the projectÕs documentation effort and have filled out several project-developed and ÒexternalÓ questionnaires and surveys online, to provide pertinent information about their opinions on project activities and happenings.

 

Most Augsburg faculty and technology advocates developed an electronic portfolio. The portfolio contains technology products that have been developed through participation in the MT2 Project. Examples of products include: PowerPoint presentations, video presentations, syllabi, digital pictures, and other technology projects. The portfolios have been utilized with varying degrees of success. Some faculty use this approach regularly, while others have found it did not meet their needs.

 

The Education Department is working to institutionalize the use of technology and online approaches. As a result of the MT2 Project, the Education Department has developed an even stronger relationship with the collegesÕ Information Technology department. IT is a key resource for online services and has welcomed input from the Education Department in technology planning for the College.

 

Analysis - Key Findings & Supporting Data

All Education Department faculty provided periodic electronic updates related to the achievement of their assigned project goals. Faculty used electronic portfolios to keep track of the project related activities/events and skill development. The Technology Advocates used electronic portfolios to maintain a record of their skill development and examples of projects completed.

Products Produced

Portfolios developed by faculty and student advocates.

 

 

Objective 6

Team Augsburg faculty members and prospective teachers with knowledgeable technology leaders in technology-rich schools both in Minnesota and around the nation, who can serve as technology and 21st Century content mentors. (GPRA: 1.1, 1.2, 1.3, 1.4, 2.1)

Definition of Success

Each faculty member will have identified a knowledgeable technology mentor and use the mentor to enhance his/her technology skills. Each Technology Advocate will participate in technology skill building sessions with knowledgeable technology leaders. Faculty and technology advocates will participate in professional meetings where technology is the primary objective.

Progress Evidence

Accomplished.

Much of the progress on this objective has been through faculty and student advocate meetings, and through attendance at local, state, and national meetings and conferences which have provided opportunities to develop a significant resource network of people, organizations, colleges and K-12 schools. Some of the meetings and conferences attended* include:

  • NECC conference (2001 & 2002) - TLT members, faculty, and Technology Advocates
  • SALT conference (2001 & 2002) - TLT members
  • APA Conference (2001) - TLT member
  • PT3 Grantees Conference (2000,2001,2002) - TLT members
  • Minnesota Association of Colleges of Teacher Education Congress (2000, 2001, 2002, 2003) - TLT members and faculty
  • MT2/PT3 Summit Group Research Conference (2001) - TLT members, faculty, K-12 Partners, and Technology Advocates
  • National School Boards Association Technology & Learning Conference (2001) - TLT members and Technology Advocates
  • TIES Technology Conference - (2000,2001,2002,2003) TLT members, faculty, K-12 partners, and Technology Advocates
  • CCTT Writing Institutes (2001 & 2002) - Faculty
  • Handhelds Conference (2002) - TLT members, faculty, K-12 Partners, and Technology Advocates
  • TIE Conference, South Dakota (2001& 2002) - TLT members and corporate partner.
  • FETC (2002) - TLT members and corporate partner.

*Attendees are required to report on conference session highlights and their implications for their teaching and learning.

 

Both faculty and student advocates had numerous opportunities to Òrub elbowsÓ with many of the stateÕs top technology leaders who have been invited to present at the project-sponsored statewide conferences, faculty retreats, and meetings of the student advocates.

Analysis - Key Findings & Supporting Data

The grant included a staff position for a knowledgeable technology leader to work one-on-one with faculty and to serve as their technology mentor. This position, filled by Bill Bierden, was invaluable in fostering an understanding and use of technology on a daily basis. Focus groups and journal entries identified the availability of one-on-one coaching to be significant in accomplishing the objectives of the grant. In addition, our partners, JDL Technologies, Inc. and TIES, afforded opportunities for mentoring by tech-savvy trainers and professional contacts. The principal and staff of Eagan High School were especially helpful in both individual mentoring to project staff, faculty and students, and in co-sponsoring events and conferences. Planning and providing opportunities such as conferences, visits and retreats, is ongoing and an effective way to continue to present the Òbig picture.Ó

Use of Evaluation Findings Products Produced

Since this objective centered on a mentoring relationship and attendance at conference/workshops there were no specific products produces. Achieving this objective provided information and skills to achieve other objectives and to produce products in other areas of the grant.

 

 

Objective 7

Disseminate and systematically share with Minnesota policy makers and teacher preparation institutions the results of the project, both during the implementation phases and after project outcomes have been validated, refined and ready for dissemination. (GPRA: 3.1, 3.2 )

Definition of Success

A dissemination plan was developed to disseminate and systematically share the results of the project. The plan will include the use of a website, presentations, publications, conferences, and other dissemination media.

Progress Evidence

Accomplished.

The most significant dissemination activity was the development of four technology monographs: Nowhere in Technology - All Children Left Behind, TalkinÕ Tall: Voices for ÒMillenniumÓ Teachers, Dreams And Shadows: Portraits Of A Professional and Harmony: One FacultyÕs Pt3 Story. These monographs provide a prospective about technology from a variety of views. The monographs are designed to stimulate discussion about technology and the role of technology in schools. The monographs were edited by Project Director, Sonja Schmieder and printed by JDL Technology, corporate partner. The monographs will be available from the Augsburg Education Department and distributed at the 2003 NECC.

 

While the dissemination of the results of this project will occur over time and after the project officially is finished, the project has disseminated information as part of an ongoing process. Augsburg faculty and grant personnel have made numerous presentations at a variety of events. Some include: PT3 Grantees Conference, CCTT Institutes, TIES conference, American Psychological Association conference, MN Association of Colleges of Teacher Education, SITE conference, and others.

In addition, the grant has developed and disseminated at project brochure and website (www .augsburg.edu/education/MT2).

Analysis - Key Findings & Supporting Data

Dissemination has been in process through this past year with the publication of the first monograph. Once the grant is completed, dissemination will be through the website and the Augsburg Education Department. The website will be maintained by Augsburg and the monograph will be available upon request. In addition, JDL Technologies will distribute the monograph upon completion of the project.

Products Produced

Grant Brochure, website, monographs.

Monograph 1: Nowhere In Technology: All Children Left Behind, published in September, 2002 - a compilation of 22 papers by top educators and technology leaders detailing the myriad of barriers we educators face in technology integration and infusion.

Monograph 2: TalkinÕ Tall: Voices For ÒMillenniumÓ TEACHERS (February, 2003) - papers by 28 authors from across the country who answered the question ÒWhat should a ÔmillenniumÕ teacher know and be able to do?Ó

Monograph 3: Dreams And Shadows: Portraits Of A Professional was written to help convey to future teachers what being a member of the teaching profession entails.

Monograph 4: Harmony: One FacultyÕs Pt3 Story. It recaps the facultyÕs journey and learning through technology-related activities and events over the course of our four-year project tenure.

A separate publication by the Augsburg College PT3 Project entitled Future Teachers: Future Schools (June, 2003) tells the story of our PT3 student technology advocates, their preparation as future leaders in both education and technology - made possible through our grantÕs weekly activities - and their perceptions of ÒmillenniumÓ schools.

 

 

Conclusions: Based on the outcomes of this project the following conclusions were identified.

1.     The Technology Leadership Team provided outstanding leadership in implementing the grant.

2.     All of the objectives of the project have been addressed and accomplished.

3.     There was a sound infrastructure in place that ensured the full achievement of the objectives of the project.

4.     Participating K-12 partners did not play a significant role in the implementation of the project. The exception was Eagan High School, which provided valuable assistance in the achievement of the objectives of the project.

5.     Corporate partner, JDL Technologies, Inc. was a valuable partner in the project.

6.     The Augsburg faculty were actively involved in developing technology skills related to using technology in the teaching and learning process.

7.     The Augsburg faculty effectively integrated technology into their methods courses.

8.     The use of the technology/curriculum specialist had a significant impact on developing technology-infused curriculum.

9.     Providing one-on-one technology mentoring to faculty was the most significant achievement of the project which lead to technology integration in teacher education courses.

10.  The use of the technology/curriculum specialist had a significant impact on student use of technology for course assignments and projects.

11.  Connecting Augsburg faculty with K-12 schools facilitates technology understanding of classroom application.

12.  Technology integration will continue to be a significant priority for the education faculty.
Appendix

 

A.               Logic Model

B.                Evaluation Worksheet

C.                Digital Dozen

D.               Syllabi Review

E.                Course Technology Survey - Day & Weekend Students

F.                Faculty Focus Group Results 2001 & 2003

G.               Advocates Focus Group Results 2002

H.               Technology Skills Survey - TAGLIT