Methods & Tools

“Evaluation methods” are the various tools that can be used to collect information for evaluation purposes. The first “featured resource” provides a quick reference guide for commonly used tools, including their advantages and disadvantages. Next, more detailed information is provided for each evaluation method.

Featured resource

Cover of the Overview of common evaluation methods resource

Watanabe, Y., & Sinicrope, C. (2008). Overview of common evaluation methods. Honolulu, HI: National Foreign Language Resource Center.

A brief list and description of common evaluation tools used in language program evaluation. Listed is each tool’s purpose, advantages, and challenges. Provided by the National Foreign Language Resource Center and the Foreign Language Program Evaluation Project.

Questionnaires

Featured

Cover of Using surveys for understanding and improving foreign language programs PDF

Davis, J. McE. (2011). Using surveys for understanding and improving foreign language programs. Honolulu, HI: National Foreign Language Resource Center.

A practical PDF guide to help language educators develop and use surveys specifically for evaluation and assessment in language programs.

Online Resources

Collecting Evaluation Data: Surveys. Short PDF guide provided by University of Wisconsin-Extension.
Assessment for Curricular Improvement, Univ. of Michigan College of Engineering. Guidelines for how to develop surveys as well as advice on choosing samples and software for data analysis.

Selected Books & Articles

Brown, J. D. (2001). Using surveys in language programs. Cambridge: Cambridge University Press.
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. Hoboken, NJ: John Wiley & Sons.
Patten, L. M. (2001). Questionnaire research: A practical guide. Los Angeles: Pyrczak.

Examples in Language Program Evaluation

Norris, J. M., Davis, J. McE., Sinicrope, C., & Watanabe, Y. (Eds.). (2009). Toward useful program evaluation in college foreign language education. Honolulu, HI: University of Hawai‘i, National Foreign Language Resource Center. [Case studies of language program evaluation projects many of which used questionnaires.]
Graduating Student Exit-Questionnaire Project at the College of Languages, Literatures, and Linguistics (CLLL), Univ. of Hawaii at Manoa. [Presentation on the project given at Georgetown University, 2013.]

Interviews

Featured

Rubin, H. J., & Rubin, I. (2012). Qualitative interviewing: The art of hearing data (3rd ed.). Thousand Oaks, CA: Sage.

A comprehensive guide on the nuts and bolts of qualitative interviewing.

Online Resources

Use of Qualitative Interviews in Evaluation. An overview of qualitative interviewing, where it is useful and how to use it, and its pros and cons from the University of Arizona.
Developing Interviews: Preparing an Interview Protocol. A useful, step-by-step professional development module by the Online Evaluation Resource Library for preparing and structuring the interview protocol.
Structured Interviews for Educational Research. A quick PARE primer on interviews, including the different types, design, and question wording and format.

Selected Books & Articles

Bourque, L. B., & Fielder, E. P. (2002). The Survey Kit (2nd ed.): Vol. 4. How to Conduct Telephone Surveys. Thousand Oaks, CA: Sage.
Gubrium, J. F., & Holstein, J. A. (Eds.). (2002). Handbook of interview research: Context & method. Thousand Oaks, CA: Sage.
Seidman, I. E. (2006). Interviewing as qualitative research: A guide for researchers in education and the social sciences (3rd ed.). New York: Teachers College Press.

Examples in Language Program Evaluation

Li, B., & Tin, T. B. (2013). Exploring the expectations and perceptions of non-native English speaking students in masters level TESOL programs. New Zealand Studies in Applied Linguistics, 19(2), 21-35.
Martinez, A., & Sanz, C. (2008). Instructors’ and administrators’ beliefs within a Spanish LSP program. In H. J. Siskin (Ed.), From thought to action: Exploring beliefs and outcomes in the foreign language program (pp. 67-91). Boston: Heinle Cengage.
Pawan, F., & Thomalla, T. G. (2006). Making the invisible visible: A responsive evaluation study of ESL and Spanish language services for immigrants in a small rural county in Indiana. TESOL Quarterly, 39(4), 683-705.

Focus Groups

Featured

Krueger, R. A., & Casey, M. A. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage.

A non-technical step-by-step guide on how to design focus group studies, select participants, moderate focus group discussion, analyze focus group data, and report results.

Online Resources

Focus Group Fundamentals. A quick PDF guide to focus groups by Iowa State University, including purposes, procedures, and reporting how-to’s. Provides a one-page table on the elements of focus groups.
Focus Groups for Evaluation. A short online reference by the University of Arizona on focus groups that touches on the pros and cons of focus groups as well as procedures and reporting.
Framework for Focus Group Use. A four-page PDF by Iowa State University that provides a framework for considering the appropriate (and inappropriate) uses of focus groups.

Selected Books & Articles

Fern, E. F. (2001). Advanced focus group research. Thousand Oaks, CA: Sage.
Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Newbury Park, CA: Sage.
Vaughn, S., Schumm, J. S. & Sinagub, J. M. (1996). Focus group interviews in education and psychology. Thousand Oaks, CA: Sage.

Examples in Language Program Evaluation

Chanda, C. (2008). Teaching and learning of English in secondary schools: A Zambian case study in improving quality. London: Commonwealth Secretariat.
Li, L., & Walsh, S. (2011). Technology uptake in Chinese EFL classes. Language Teaching Research, 15, 99-125.
Milleret, M. (2008). The trials and tribulations of comprehensive program evaluation. ADFL Bulletin, 39(2 & 3), 44-48.

Document Analysis

Featured

Cover of CDC evaluation brief on document review

Centers for Disease Control and Prevention (CDC), (2009). Data collection methods for evaluation: Document review (No. 18).

A two-page CDC evaluation brief for conducting and planning document review for evaluation purposes. Provides a brief introduction to document review and an overview of its advantages and disadvantages.

Online Resources

Document Analysis. A short overview of document analysis, its distinctive features, and key document-analysis questions by SAGE.

Selected Books & Articles

Caulley, D. N. (1983). Document analysis in program evaluation. Evaluation and Program Planning, 6(1), 19-29.
Patton, M. Q. (2002). Qualitative research & evaluation methods. Thousand Oaks, CA: Sage. (see chapter 6 on field work strategies and observation methods).
Prior, L. (2003) Using Documents in Social Research. London: Sage.

Examples in Language Program Evaluation

Peacock, M. (2009). The evaluation of foreign-language-teacher education programmes. Language Teaching Research, 13(3), 259-278.
Watanabe, Y. (2006). A Needs analysis for a Japanese high school EFL general education curriculum. Second Language Studies, 25(1). Retrieved from http://www.hawaii.edu/sls/wp-content/uploads/2014/09/Watanabe.pdf

Observation

Featured

Wajnryb, R. (1992). Classroom observation tasks: A resource book for language teachers and trainers. New York, NY: Cambridge University Press.

A how-to guide to observations in language classrooms. It provides a range of practical, hands-on tasks that guide the user through the processes of observing, analyzing, and reflecting.

Online Resources

Collecting Evaluation Data: Direct Observation. A PDF guide from the University of Wisconsin Extension on the values, types (structured and unstructured), instrumentation, and data analysis of observation. Example observation guides and recording sheets are useful.
Classroom Observation Instruments. A set of worksheets, forms, reports, and checklists for various types of observation, including peer, faculty, and classroom observations from the University of Minnesota.
Practitioner’s Guide to Conducting Classroom Observations. A series of PDF guides by the University of Virginia that provides snapshots of why to use classroom observations, how to go about selecting and using classroom observation tools, and how they can be used to support the systematic improvement of teacher effectiveness.

Selected Books & Articles

Anderson, J. (1998). Managing and evaluating change: The case of teacher appraisal. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 159-186). London: Longman.
Coombe, C., Al-Hamly, M., Davidson, P., & Troudi, S. (Eds.). (2007). Evaluating teacher effectiveness in ESL/EFL contexts. Ann Arbor, MI: The University of Michigan Press.
Lofland, J., Snow, D. A., Anderson, L., & Lofland, L. H. (2005). Analyzing social settings: a guide to qualitative observation and analysis (4th ed.). Belmont, CA: Wadsworth.

Examples in Language Program Evaluation

Griva, E., & Sivropoulou, R. (2009). Implementation and evaluation of an early foreign language learning project in kindergarten. Early Childhood Education Journal, 37 (1), 79-87.
Koyalan, A. (2009). The evaluation of a self-access centre: A useful addition to class-based teaching? System, 37, 731-740.
Rea-Dickins, P. (2001). Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing, 18(4), 429-462.

Needs Analysis

Featured

Huhta, M., Vogt, K., & Tulkki, H. (2013). Needs analysis for language course design: a holistic approach to ESP. Cambridge University Press.

A toolkit for teaching professionals who are looking to design language courses for learners with professional or vocational goals or interests.

Online Resources

One Stop English: Needs Analysis. Alex Case compliments a six-part guide to assessing needs for specific purposes (business English in this case) with worksheets and lesson plans.
Needs Analysis: How to determine training needs. This website by HR-Guide.com targets training needs for employment. Various types of analysis are briefly explained (e.g., context analysis, user analysis, cost-benefit analysis).

Selected Books & Articles

Altschuld, J. W., & Watkins, R. (2014). A primer on needs assessment: More than 40 years of research and practice. New Directions for Evaluation, 144, 5-18.
Brown, J. D. (2009). Foreign and second language needs analysis. In M. H. Long & C. J. Doughty (Eds.), The handbook of language teaching (pp. 269-293). Oxford: Blackwell.
Long, M. H. (Ed.) (2005b). Second language needs analysis. Cambridge: Cambridge University Press.

Examples in Language Program Evaluation

Iwai, T., Kondo, K., Lim, D. S. J., Ray, G., Shimizu, H., & Brown, J. D. (1999). Japanese language needs analysis 1998-1999 (NFLRC NetWork #13) [PDF document]. Honolulu, HI: University of Hawai‘i, Second Language Teaching & Curriculum Center.
Lett, J. A. (2005). Foreign language needs assessment in the US military. In M. H. Long (Ed.), Second language needs analysis (pp. 105-124). Cambridge: Cambridge University Press.
Van Avermaet, P., & Gysen, S. (2006). From needs to tasks: Language learning needs in a task-based approach. In K. Van den Branden (Ed.), Task-based language teaching in practice (pp. 17-46). Cambridge: Cambridge University Press.