Evaluation of Machine Translation systems


Idoia Martínez del Mozo

Ángela Maside Páramo

Alejandro Otaola Rojo




This report is an assigned investigation task for our class of "English Language and New Technologies". This is a subject included in the second year of the English Philology degree in the University of Deusto (Universidad de Deusto), in Bilbao (Spain). This subject is taught by Professor Joseba Koldobika Abaitua Odriozola, and this part of the subject deals with the evaluation of Machine Translation systems which exist nowadays.

This is the second report we have been asked to elaborate during this course on New Technologies. As during this second half of the course we have been dealing with Machine Translation, we are now developing the knowledge that we have acquaired on that, by evaluating some machine translators that can be found in the internet.



In this second report of this course on New Technologies, we are dealing with Machine Translation and in a more accurate way, with machine translators that can easily be found in the internet. These translators may be useful tools in order to make a simple translation in a moment of hurry or urgent necesity, but we do not think they constitute a totally reliable tool in the field of Machine Translation.

What we will try to do with this report is mainly a task of evaluation of three different machine translators that are found in the internet. And the methodology that we have chosen so as to make that evaluation of the different translators is that of comparing them directly by translating the same "message" or "piece of information" from one language into another through the machine translators that are to be analysed in this report.

Although the translators that can be found on the net have the possibility of translating from many languages into a great number of them, we will focus our analysis mainly on the translation from English into Spanish and vice versa.

For that purpose, we will use the following machine translators:





And to complete the comparative analysis of the translators, we will provide the linguistically correct and appropriate version of each translation with the help of a paper dictionary English-Spanish/Spanish-English.


Useful information on Machine Translation (MT)

The term machine translation (MT) is normally taken in its restricted and precise meaning of fully automatic translation. However, in this chapter we consider the whole range of tools that may support translation and document production in general, which is especially important when considering the integration of other language processing techniques and resources with MT We therefore define Machine Translation to include any computer-based process that transforms (or helps a user to transform) written text from one human language into another. We define Fully Automated Machine Translation (FAMT) to be MT performed without the intervention of a human being during the process. Human-Assisted Machine Translation (HAMT) is the style of translation in which a computer system does most of the translation, appealing in case of difficulty to a (mono- or bilingual) human for help. Machine-Aided Translation (MAT) is the style of translation in which a human does most of the work but uses one of more computer systems, mainly as resources such as dictionaries and spelling checkers, as assistants.

There were of course dissenters from the dominant 'perfectionism'. Researchers at Georgetown University and IBM were working towards the first operational systems, and they accepted the long-term limitations of MT in the production of usable translations. More influential was the well-known dissent of Bar-Hillel. In 1960, he published a survey of MT research at the time which was highly critical of the theory-based projects, particularly those investigating interlingua approaches, and which included his demonstration of the non-feasibility of fully automatic high quality translation (FAHQT) in principle. Instead, Bar-Hillel advocated the development of systems specifically designed on the basis of what he called 'man-machine symbiosis', a view which he had first proposed nearly ten years before when MT was still in its infancy (Bar-Hillel 1951).

Nevertheless, the main thrust of research was based on the explicit or implicit assumption that the aim of MT must be fully automatic systems producing translations at least as good as those made by human translators. The current operational systems were regarded as temporary solutions to be superseded in the near future. There was virtually no serious consideration of how 'less than perfect' MT could be used effectively and economically in practice. Even more damaging was the almost total neglect of the expertise of professional translators, who naturally became anxious and antagonistic. They foresaw the loss of their jobs, since this is what many MT researchers themselves believed was inevitable.

In these circumstances it is not surprising that the Automatic Language Processing Advisory Committee (ALPAC) set up by the US sponsors of research found that MT had failed by its own criteria, since by the mid 1960s there were clearly no fully automatic systems capable of good quality translation and there was little prospect of such systems in the near future. MT research had not looked at the economic use of existing 'less than perfect' systems, and it had disregarded the needs of translators for computer-based aids.

While the ALPAC report brought to an end many MT projects, it did not banish the public perception of MT research as essentially the search for fully automatic solutions. The subsequent history of MT is in part the story of how these is this mistaken emphasis of the early years has had to be repaired and corrected. The neglect of the translation profession has been made good eventually by the provision of translation tools and translator workstations. MT research has turned increasingly to the development of realistic practical MT systems where the necessity for human involvement at different stages of the process is fully accepted as an integral component of their design architecture. And 'pure' MT research has by and large recognised its role within the broader contexts of commercial and industrial realities.

Ten years ago, the typical users of machine translation were large organizations such as the European Commission, the US Government, the Pan American Health Organization, Xerox, Fujitsu, etc. Fewer small companies or freelance translators used MT, although translation tools such as online dictionaries were becoming more popular. However, ongoing commercial successes in Europe, Asia, and North America continued to illustrate that, despite imperfect levels of achievement, the levels of quality being produced by FAMT and HAMT systems did address some users’ real needs. Systems were being produced and sold by companies such as Fujitsu, NEC, Hitachi, and others in Japan, Siemens and others in Europe, and Systran, Globalink, and Logos in North America (not to mentioned the unprecedented growth of cheap, rather simple MT assistant tools such as PowerTranslator).


Several applications have proven to be able to work effectively using only subsets of the knowledge required for MT. It is possible now to evaluate different tasks, to measure the information involved in solving them, and to identify the most efficient techniques for a given task. Thus, we must face the decomposition of monolithic systems, and to start talking about hybridization, engineering, architectural changes, shared modules, etc. It is important when identifying tasks to evaluate linguistic information in terms of what is generalizable, and thus a good candidate for traditional parsing techniques (argument structure of a transitive verb in active voice?), and what is idiosyncratic (what about collocations?). Besides, one cannot discard the power of efficient techniques that yield better results than older approaches, as illustrated clearly by part of speech disambiguation, which has proved to be better solved using Hidden Markov Models than traditional parsers. On the other hand, it has been proven that good theoretically motivated and linguistically driven tagging label sets improve the accuracy of statistical systems. Hence we must be ready to separate the knowledge we want to represent from the techniques/formalisms that have to process it


Within the last ten years, research on spoken translation has developed into a major focus of MT activity. Of course, the idea or dream of translating the spoken word automatically was present from the beginning (Locke 1955), but it has remained a dream until now. Research projects such as those at ATR, CMU and on the Verbmobil project in Germany are ambitious. But they do not make the mistake of attempting to build all-purpose systems. The constraints and limitations are clearly defined by definition of domains, sublanguages and categories of users. That lesson has been learnt. The potential benefits even if success is only partial are clear for all to see, and it is a reflection of the standing of MT in general and a sign that it is no longer suffering from old perceptions that such ambitious projects can receive funding.


Machine Translation (MT) is a possible way of overcoming language barriers. However output quality is often a problem. Let us see why.

We will consider some particular problems which the task of translation poses for the builder of MT systems --- some of the reasons why MT is hard. It is useful to think of these problems under two headings: (i) Problems of ambiguity, (ii) problems that arise from structural and lexical differences between languages and (iii) multiword units like idioms and collocations. We will discuss typical problems of ambiguity in Section, lexical and structural mismatches in Section, and multiword units in Section .

Of course, these sorts of problem are not the only reasons why MT is hard. Other problems include the sheer size of the undertaking, as indicated by the number of rules and dictionary entries that a realistic system will need, and the fact that there are many constructions whose grammar is poorly understood, in the sense that it is not clear how they should be represented, or what rules should be used to describe them. This is the case even for English, which has been extensively studied, and for which there are detailed descriptions -- both traditional `descriptive' and theoretically sophisticated -- some of which are written with computational usability in mind. It is an even worse problem for other languages. Moreover, even where there is a reasonable description of a phenomenon or construction, producing a description which is sufficiently precise to be used by an automatic system raises non-trivial problems.


Evaluation of machine translators (SYSTRAN, REVERSO and PROMT)

Colloquial Expressions

In this table we can see how four different translators including the dictionary, translate some Spanish colloquial expressions into English, without attending to the general context, because they translate word by word, except in the dictionary translations.






 Contra viento y marea

Against all odds

Against wind and tide

Against wind and tide

Fight against all the odds to save it

Estar en una nube To be in a cloud To be in a cloud To be in a cloud

To be daydreaming

A freír espárragos

To freir asparagus

To freir asparagus

To frying asparagus

To tell somebody to get lost

Llevar al huerto

To take to the orchard

To lead to the garden

To lead to the orchard

To wicked way with somebody

Estar como una vaca

To be like a cow

To be as a cow

To be as a cow

To be very fat

Perder los papeles

to lose the papers

To lose the papers(roles)

to lose the roles

To lose one's touch

Estar como una cabra

To be like a goat

To be as a goat to be as a goat

To be nuts

No ver tres en un burro

not to see three in a donkey Not to see three in a donkey not to see three in a donkey

To be as blind as a bat

Estar loco de atar

To be crazy to tie Madman of tying Madman of tying

To be nutty as a fruit cake

Somos uña y carne

To be nail and meat To be a fingernail and meat To be a fingernail and meat

To be as thick as thieves


Other Expressions

In this table we can not see very common colloquial expressions, translated by several machine translators, from Spanish to English. From this translations, it is easy to see that this expressions in Spanish,then translated to english, do not vary so much.


Porque me da la gana Because it gives the desire me Because it(he,she) gives me the desire because it gives me the desire Because I want to
Porque sí Because yes Because yes because yes Because
No seas picajoso You are not picajoso Do not be picajoso do not be peevish Don't be so touchy
No seas cabezota You are not big head Do not be a large-headed person do not be a large-headed person Don't be pigheaded
Es broma it is joke It is a joke it is a prank Just joking
Sano y salvo Safe and sound I recover and save healthy and safe safe and sound


Orthographic Problems

In the case of this paper, in which we are dealing with translation from Spanish into English, an added problem to the translation is the existence of the accent in Spanish. As English does not have a graphic mark of accent, the translator has to be capable of interpreting the function of the accent in Spanish words.

Another important feature related to this, is the existence of  the diacritic accent in Spanish, which means that two or even three words can be differenciated in meaning by the use of that accent. Although the letters of the two or three words and their order are exactly the same, the only way to differenciate the meanings is the diacritic accent, which is a fundamental sign in Spanish spelling.


¿Está en casa? It is in house? Is it(he,she) in house? Is it in house? Is he/she at home?
¿Esta en casa? This in house? This one in house? This one in house? ----------
El té de las cinco The tea of the five The tea of the five the tea of the five Five o'clock tea (*)
El te de las cinco You of the five You of the five you of the five ----------
Soy de Bilbao I am of Bilbao I am from Bilbao I am from Bilbao I'm from Bilbao
Soy dé Bilbao I am gives Bilbao I am give Bilbao I am give Bilbao ----------

(*) As we were unable to find a translation in the dictionary that has been used by us for this paper, we have been obliged to ask an informant of English. In this particular case, the informant has been Professor Peter Lavery, who belongs to the English Department of the University of Deusto, and teaches English Linguistics courses among other subjects.


Words With Different Meaning

There are many words in both English and Spanish that have many different meanings depending on the differents contexts in which they appear. When you use machine translation to translate this kind of word, you find that the translator only gives you one of the possible meanings of the word, usually the most common use. This can be clearly seen in the table below:

Fair Justo Feria Feria Justo, (pelo) rubio, feria
Table Tabla Mesa
Mesa Mesa, tabla, presentar
Sole Único Exclusivo
Suela Lenguado, único, suela
Mean Medio Tacaño Medio Tacaño, media/a, significar
Book Libro Libro Libro Libro, reservar, multar
Cola tail Tail Tail Tail, glue, queue
Banco Bank Bank Bank Bank, bench, shoal
Sitio site Site(Place) Place Place, siege
Franco franc Franc Franc Frank, franc
Botín booty Booty Booty Ankle boot, loot


Idioms In English

Another problematic field in machine translation is idioms. An idiom is a group of words that have a different meaning from the usual meaning of the separate words.The problem in machine translation translating idioms is that the translators give the meaning of each separate word. As a result we get a non-sensical sentence that has nothing to do with the real meaning of the idiom. In the table below we can see some examples of English idioms translated into Spanish:

It's rainning cats and dogs Está lloviendo gatos y perros Esto es gatos rainning y perros Es gatos rainning y perros Llueve a cántaros
Don't count your chickens until they hatch No cuente sus pollos hasta que traman No cuente sus pollos hasta que ellos incuben No hay que vender la piel del oso hasta que ellos incuben No vendas la piel del oso antes de cazarlo.
Out of the question Inadmisible Inadmisible Inadmisible Imposible/Impensable
To rub someone the wrong way Para frotar a alguien la manera incorrecta Frotar a alguien el camino incorrecto Frotar a alguien el camino incorrecto Fastidiar a alguien
Zip your lip Relampague su labio Pase rápidamente (Cierre la cremallera) su labio Cierre la cremallera de su labio Cierra el pico
To drag one's feet Para arrastrar sus pies Arrastrar pies de alguien Arrastrar pies de alguien Dar largas

English Idioms taken from: http://english-zone.com/idioms/idioms.php?CID=33


Phrasal Verbs

A phrasal verb is a verb followed by a preposition or complement, which gives a complete new meaning to the verb. So it means that the meaning of the phrasal verb is not the addition of the main verb's meaning and the preposition or complement's meaning.

That problem constitutes the most difficult task for a machine translator, because it ought to be capable of differenciating the meanings according to the preposition that each verb takes.







Give up

Dé para arriba



Renunciar/ Entregarse

Look  for




Desear que llegue algo.

Get down

Consiga abajo



Bajar/ Reducir

Look after





Go out





Break down




Estropearse/ Derribar

Look up

 Mire para arriba

Alzar la vista

 Alzar la vista

Buscar/ Mejorar


 Idioms In Spanish

In this table we can see some Spanish idioms translated into English through different machine translators.






Más vale pájaro en mano que ciento volando.

More bond bird in hand,that one hundred flying.

More it cost bird in hand, which hundred demolishing.

More it costs bird in hand, that hundred flying.

A bird in the hand is worth two in the bush

A caballo regalado no le mires el diente.

At given horse you do not watch the tooth to him

Astride given look at the tooth.

Astride given look at the tooth for him

Don´t look a gift horse in the mouth.

Ojos que no ven, corazón que no siente.

Eyes that do not see, heart that does not feel.

Out of sight, out of mind.

Eyes that they do not see, heart that he does not feel

Out of sight ,out of mind.

Cría cuervos y te sacarán los ojos.

It raises crows and they will remove the eyes to you

It(He,She) raises ravens and the eyes will extract you

It raises ravens and the eyes will extract you

That's the thanks you get

Hasta el cuarenta de mayo no te quites el sayo.

Until the forty of May you do not take off sayo.

Until May, forty do not take the sayo from yourself

Until fortieth of May do not take the sayo from yourself

Never cast a clout till may is out, you can´t be sure of warm weather until June


As we can see from this paper, there are several interpretations for the different aspects of language. These interpretations are given by three different translators and a dictionary. After comparing the translation given by the three machine translators are different among them in many cases, what indicates that they all cannot be correct. 

The purpose of the translations taken from the dictionary has been that of using it so as  to check the correction of the translations provided by the three different machine translators. 

We have realized with this paper is that those machine translators that can be found in the internet do not provide an accurate translation for the phrases that are to be translated. Many factors have influence upon this difference of translations from the same phrase.

We can claim that although the machine translators may be a useful tool for translating some sorts of pieces of information, they are not accurate enough so as to make a reliable and serious translation. They are unable to provide a good translation in many cases, so the figure of the translator and the philologist is essential for translating.

To sum up, we can say that although they might help, they are not enough for what language translation needs nowadays. We, as experts of Language, are essential for translation, although we appreciate the help provided by machine translators.



Web page for SYSTRAN translator: http://www.systranbox.com/  

Web page for REVERSO translator: http://elmundo.reverso.net/textonly/default.asp  

Web page for PROMT translator: http://translation2.paralink.com/