Translation Program Evaluation

Volume 5, No. 1
January 2001

Moustafa Gabr was born in Cairo, Egypt, in 1964. He obtained his BA degree in English Language and Literature in 1987 at Ain Shams University in Cairo. He also received his advanced Translation Certificate in Translation and Interpreting from the American University in Cairo in the same year. In 1991, he obtained his Graduate Translation Diploma at Cairo University. Gabr was awarded his MA degree in translation pedagogy in December 2000.

Gabr has 14 years of diverse experiences in translation, interpreting and editing in world-class organizations in Egypt, Saudi Arabia and Kuwait. He has been involved in translator/interpreter training and curriculum development since 1991. He participated in several local and international seminars on translation theory and practice. He has several articles and books on translation and translator training published under his name. Under the auspices of Oxford University Press, UK, he has produced the English-Arabic version of the Oxford-Duden English Pictorial dictionary, which will be published in April 2001. Currently, Gabr is the Translation Department Manager in Achievement for Consultancy and Training (ACT), a private-sector training company in Kuwait. Moustafa Gabr can be reached at moustafa@consultant.com.

	From the Editor
	TJ in the New Millennium by Gabe Bokor
	Index 1997-2001
	Translator Profiles
	A Vocation Found by Danilo Nogueira
	The Profession
	The Bottom Line by Fire Ant & Worker Bee
	Existe-t-il des traducteurs spécialisés ? by Frédéric Houbert
	Translator Education
	Revision—Food for Thought by Nick Somers
	Program Evaluation—A Missing Critical Link in Translator Training by Moustafa Gabr
	Spanish
	Strategies for Translating "-ing" Adjectives and Nouns from English to Spanish by Constanza Gerding Salas
	Portuguese
	World Translation Contest by Danilo Nogueira
	Interpretation
	The Importance of Knowing Your Local Vernacular by Serghei G. Nikolayev, Ph.D.
	Advertising Translation
	Translation Practices in International Advertising by Mathieu Guidère, Ph.D.
	Science & Technology
	A Translator’s Guide to Organic Chemical Nomenclature XXII by Chester E. Claff, Jr., Ph.D.
	Dictionary Reviews
	The Very Mind of English?—Activate Your Mind, Socrates Style by Zsuzsanna Ardó
	The Very Heart of English?— Reflections on culture, fluency and the native speaker's head prompted by the LDELC by Zsuzsanna Ardó
	Two New Portuguese Dictionaries by Gabe Bokor
	Translators and Computers
	Machine Translation and the Future of the Translation Industry by Yves Champollion
	From Novelty to Ubiquity: Computers and Translation at the Close of the Industrial Age by Celia Rico Pérez
	Caught in the Web
	Web Surfing for Fun and Profit by Cathy Flick, Ph.D.
	Translators’ On-Line Resources by Gabe Bokor
	Translators’ Tools
	Using MS Word's Advanced Find and Replace Function by Tibor Környei
	Translators’ Emporium
	Letters to the Editor
	Translators’ Events
	Call for Papers and Editorial Policies

Program Evaluation

A Missing Critical Link in Translator Training

by Moustafa Gabr

Introduction

ranslation, being a craft on the one hand, requires training, i.e. practice under supervision, and being a science on the other hand, has to be based on language theories. Therefore, any sound approach to translation teaching has to draw on proper training methodologies. Training focuses on the improvement of the knowledge, skills and abilities of the individual, and it is functional and relevant only when it is evaluated (Zenger and Hargis, 1982). When we evaluate a training course, we actually evaluate its effectiveness, i.e. we measure the achievement of its objectives. A training course can be effective in meeting some objectives and be ineffective in meeting others. For example, a translation course may accomplish its objective of improving the the students' text analysis skills and fail in promoting their cross-cultural awareness.

When we evaluate a training course, we actually evaluate its effectiveness, i.e. we measure the achievement of its objectives.

It is expected that translation courses be carefully evaluated on a regular basis. But what happens in reality is that educational institutions hardly pay attention to translation course evaluation, and if the evaluation step is considered at all, it is traditionally placed at the end of the course. As a result, it tends to be skimped, or even neglected, and the form of evaluation commonly used at this point measures only student reaction, which is useful for but a few of the decisions that must be made about the course.

Translation course evaluation is not done more frequently for several reasons. First, there is a tendency to assume that "(a translation) training (course), being a form of education, is a good thing and therefore its evaluation is not necessary" (Sheal, 1990). Moreover, those involved in translation course design and implementation may tend to be afraid of criticism or even of the assumption that they could be replaced in case it is established that the course is not effective. They may also assume that they fulfilled their evaluation responsibility in their pre-course phase. Their reasoning is that they would not prepare a course that they did not think was going to work, hence if they have made prudent decisions on the course content and delivery beforehand, then it is not necessary to conduct course evaluation. They tend to forget that many courses were designed under the assumption that they were almost perfect, and after application they proved to be a failure. Another reason is that the evaluation process itself is complex and requires time, effort, and expertise. These resources may not be available or those involved in course design and implementation may not be willing to take the trouble to expend them. Moreover, those involved in translation course design and implementation are either professionals or academics. They are not training specialists, and hence they do not look at the training system from the right perspective.

Definition of evaluation

Notably, a great deal of attention has been given to the issue of how a translation course participant should be evaluated and how the quality of a translation can be rated whereas the evaluation of the course itself has hardly received any attention despite its importance. Training evaluation is defined as "the systematic collection of descriptive and judgement information necessary to make effective training decisions related to the selection, adoption, value, and modification of various instructional activities" (Goldstein, 1980, p.237). Descriptive information on the one hand provides an idea about what is happening or has happened, while judgement information reflects some opinion or belief about what has happened. For instance, a student may comment that "the instructor encouraged questions". This comment contains descriptive information; it simply states a fact. The student's other comment "the instructor did not answer the questions adequately" provides judgement information—the student's opinion based on the fact. Both descriptive and judgement information are required for effective evaluation of the translation course.

Functions of evaluation

In general terms, evaluation provides a guide for future improvements. More importantly, if it is instituted as an integral and continuing part of the training processes, it can provide an early warning of deficiencies and the chance for many on-the-spot improvements. There are four main inter-related functions of translation course evaluation:

Determining whether the processes of learning and transfer of knowledge have been successful, i.e. whether the course is accomplishing its objectives;

Determining whether the objectives, course content and delivery are appropriate to the process of learning and skills transfer needs;

Identifying the strengths to be maintained and the weaknesses that need to be addressed, thus helping to improve the quality of current and future courses.

Determining whether the course should be continued, revised or even discontinued.

In brief, it can be concluded that evaluation helps to determine whether or not objectives are being (or have been) met, and permits problem areas that require development to be identified. Evaluation studies show that feedback on the effectiveness of training programs does improve program content and instructor performance.

Methods of evaluation

There are several methods for course evaluation. The most effective and commonly used are the following three tools:

Feedback from students

Being the actual receivers of training, the students should be involved in evaluating the course so that the appropriateness of the training styles and methods, on the one hand, and meeting course objectives, on the other, can be determined. This involves both evaluation at the student reaction level and evaluation at the student learning level as explained below.
1. Evaluation at the student reaction level
  
  This kind of feedback can be obtained at the conclusion of the training sessions. Evaluating the course at its end examines the students' reaction to the training. In evaluating the course at this level, the focus is on the students' perception about the course and its effectiveness. This information is important for the continuity of the course. Nevertheless, this information cannot indicate whether the course has met its objectives beyond ensuring student satisfaction. (Harris and DeSimone, 1994). Evaluating the course at this point is best achieved through completing a questionnaire. The questionnaire method is the most popular instrument for post-course evaluation because it has many advantages. It is the best tool to be used for large numbers of respondents; it provides quantitative data for analysis; it gathers in-depth information on the training students need; it can be completed and analyzed quickly; it is relatively inexpensive; it is more accurate if anonymous and is convenient, because the respondent sets the pace; it also provides a variety of response options and thus is easier to answer. For evaluating a translation course, the questionnaire can address such issues as:
  1. Timeliness of the course
  2. Length of the course
  3. The appropriateness of the course objectives to the actual needs of students
  4. The relevance of material presented and any areas that the students think need more attention
  5. Quality of the material
  6. The appropriateness of the methods and styles of training delivery used—which ones were effective in imparting information to the students and which ones were not (for example lecturing, discussion, audio-visual aids, etc.)
  7. The appropriateness of the learning environment
  8. The appropriateness of student evaluation tools and policy
  9. The efficiency of the roles played by the instructor
  10. The overall performance of the course
  11. The overall performance of the instructor
  12. Achievement of the course objectives
  13. Relevance of the course to market needs
  14. Suggestions/comments about course content
  15. Suggestions/comments about training delivery
  16. Suggestions/comments about instructor's style
  17. Other suggestions/comments the students may have
  Tips on feedback form preparation
  
  Following are some tips on preparing a post-course evaluation questionnaire:
  1. Determine clearly with the other individuals involved in the training course what you wish to know and why you wish to know it.
  2. Decide at the questionnaire design stage how the data is going to be analyzed and used.
  3. Try to involve other instructors in designing the form to benefit from their perspectives and ideas.
  4. Prepare a standard form, stating specific objectives and conditions for training. Remember that when the questionnaire is more relevant, it is easier and more likely to be completed.
  5. Personal data of the student should always be left optional to avoid overly positive feedback.
  6. Use a five-point scale of satisfaction to provide more meaningful gradations; avoid using only three (3) levels of ratings.
  7. Make sure that the questionnaire is easy to read, easy to answer, and provides information that can be used for evaluating the effectiveness of the course.
  8. Use a mixture of forced choice and open-ended items, i.e. items which provide the respondent with a series of options from which they must choose, while the latter does not narrow the range of possible responses.
  9. Match the type of response to the question.
  10. Do not ask respondent to answer more than one statement at a time. In other words, avoid the so-called double-barreled items.
  11. Emphasize key words in each item by using bold letters, italics, or underline.
  12. Have the student rate and comment on both the conditions and content of the training.
  13. Balance the questions between course content and delivery.
  14. Ask for overall ratings for both the course and the instructor.
  15. Do not keep the respondent switching from one topic to another and back again. You may prepare a group of statements/questions on the instructor, another group on the course content, a third on the learning environment, etc.
  16. Balance questions that have positive and negative biases.
  17. Allocate time on the agenda for completing the questionnaire by the students. It is extremely difficult to collect questionnaires after the course.
  18. Encourage the students to give suggestions about ideal resources and conditions for training implementation.
  19. Change the order of questions on the questionnaires from course to course, and customize the content in light of the course evaluated.
  20. Ensure that the students view this testing as a tool for evaluating the course, not the students.
  21. Test the questionnaire on a pilot sample to ensure that it is clearly understood. This will also help you determine how long it will take to answer the questionnaire, eliminate any ambiguities in the statements, and evaluate how much flexibility there is for comments.
  It is recommended that the students be given feedback by the instructor as to how their suggestions and comments have been analyzed and what future improvements, if any, will be implemented. This will accentuate the importance attached to their opinions.
2. Evaluation at the student learning level
  
  Another way for evaluating the effectiveness of the course is to evaluate the progress of the students at certain intervals. It involves the evaluation of the skills and knowledge the students have gained during the course. It requires a systematic evaluation plan to measure the students' learning, and whether the objectives of the course were met. In this case, pre-course tests may be used to gauge the level of skills and knowledge before training. Once the course commences, periodic tests (or a mid-course test) are to be held to assess the development of the students' acquired skills and knowledge in light of the objectives covered in the period prior to each test. Finally, a comprehensive post-course test should be designed to determine the new levels of skills and knowledge gained by the students. If there is an marked overall improvement, then the course is successful in terms of meeting its objectives. If not, the appropriateness of the course has to be questioned. This technique, alone, evaluates the students' understanding of the key learning points and thus the appropriateness of the course to developing their skills and knowledge. It is inadequate in that it does not identify, in terms of content and delivery, the weaknesses of the course, and thus does not provide indicators for current or future course improvements.
Feedback from instructors

Monitoring group activities and the knowledge manifested during class discussions and assessing them against the session objectives.

Home assignments that can be later collected and assessed by the instructor in light of what was taught.

Completion of an evaluation form by the instructor after each session, or on a weekly basis, in which the instructor records his observations on the relevance of the material to the objectives covered in the session, appropriateness of the teaching method(s) and aids used, and other class facilities.

Monitoring how the students perform in the final exam. It should be noted however that this method should not be used as the one and only tool of evaluation. Waiting until the final exam to assess whether the students have acquired sufficient knowledge and skills may be useful in terms of assessing the overall performance of the course. But it is likely to fail in identifying the defects in content and delivery that require immediate attention as the course progresses.

Engaging actively in continual evaluation communication with the students, for instance during breaks or at the beginning or end of each session, to identify where knowledge gaps still exist and the students' preferences for presentation styles.

Observation by independent evaluators

Independent evaluators, such as department heads, program managers, or coordinators, can also play a significant role in the evaluation process. They can identify where improvements, particularly in the delivery style and class facilities, are needed. As such, an evaluator pays a visit to the class and writes down his observations on the teaching methods adopted, rapport with the students, instructor's knowledge of the subject matter, time management and adherence to course objectives, appropriateness of the learning environment to the transfer of knowledge, etc. Then, the evaluator and the instructor meet to discuss these observations. They agree on areas for improvement and jointly work out a plan for development. Finally, the evaluator follows up the execution of this plan by the instructor to ensure that deficiencies are properly addressed.

Conclusion

Training processes, being a collective effort aiming at improving the individual performance of the student to qualify him/her to join the profession, should be viewed as a collective undertaking that requires close cooperation, coordination and meticulous evaluation by all parties involved in the training. In order for a training course to bear fruit, it has to be monitored, i.e. subjected to evaluation. The evaluation process may be conducted, as previously mentioned, either at intervals or at the end of the course. To achieve the goals of evaluation, it is recommended that the evaluation process be conducted at the pre-course phase, at each step in the training cycle and at the end of the course. Thus, constructive feedback can be obtained and utilized in a timely manner. I have designed the following model shown below, which I call the Comprehensive Quality Control Model (CQCM). It ensures accuracy in designing and implementing each step in the training effort and, eventually, a quality product. It comprises the three levels of evaluation as follows:

Evaluation at the training development level

Evaluation at the student learning level

Evaluation at the student reaction level

References

Charney, C. and Conway, K. (1996), The Trainer's Tool Kit, New York, NY: AMACOM.

Goldstein, I.L. (1980). Training in Work Organizations. Annual Review of Psychology, 31, 229 - 272.

Harris, D. M. and DeSimone, R.L. (1994). Human Resources Development, Orlando, Fl: the Dryden Press.

Phillips, J.J. (1983). Handbook of Training Evaluation and Measurement Methods, Houston, TX: Gulf.

Sheal, P.R. (1990). How to Develop and Present Staff Training Courses, London: Kogan Page Limited.

Zenger, J.H. and Hargis, K. (1982). Assessment of Training Results: It's Time to Take the Plunge! Training and Development Journal, 36, (1), 11 - 16.

Comprehensive Quality Control Model (CQCM)