Levels & Labels

A service level must be agreed on before production begins. This is called pre-production and includes project initiation in 11669. The following tasks are found at least implicitly in both 11669 and F2575.

– Describe the Use Case (in F2575, section 5 “Needs Analysis”, a use case consists of five things: subject field, type of text, topic, audience, and purpose. Implicitly, a use case also includes source language and target language(s)). There is a use case appendix at the end of this document, including hospital instructions to a patient and pre-trial triage.

– Conduct a Risk Assessment (which consists of risk identification, analysis, and evaluation according to ISO 31000) (What risks are associated with the use case? E.g., consider operating instructions for a crane that is one hundred meters tall. An incorrect translation could result in severe damage to a building.)

– Decide on Risk Tolerance Level for Errors in Correspondence and Fluency (which can result in risk mitigation activities to be conducted during production or associated with production, such as a bilingual comparison of source and target content)

– Select a Method of Production (from fully automatic to fully human) compatible with the Risk Tolerance Level [method of delivery, from fully automatic to fully human and from remote to in person]

– Agree on all other relevant service requirements for the use case (that is, requirements beyond risk tolerance level, risk mitigation activities, and method of production). Additional service requirements are linked to parameters such as compensation and delivery deadline. See the “translation parameters” section of 11669 or F2575 for the full list of parameters, which are basically the same in the two standards. For an abbreviated list of those parameters, see the Translation Parameters appendix at the end of this document. Parameters, in F2575, are standard questions; specifications are the answers in a particular use case.

Note 1: An SLA as conceived in these translation standards includes preliminaries that might be assumed rather than explicit in a typical SLA. Assumptions can lead to potential misunderstanding and conflict between stakeholders, whereas clear specifications can reduce both misunderstanding and conflict.

Note 2: The order of these five components of the pre-production phase is not random. Each builds logically on the previous one in a completed description of a multilingual language service, even though real-life conversations between a requester and a provider or advisor might start with any of them.

Once the requester and provider have covered the five aspects of a project [encounter rather than project] in part A (pre-production), they have defined a service level. A distinguishing feature between service levels in the world of translation is risk tolerance level for errors, but each service level includes much more than risk tolerance level.

The service level that has been agreed on should be documented whether we are dealing with translation or interpreting. One example of documentation is a service level agreement. Service level agreements can be broad or they can be specific to a particular client. 

Here is an example of an agreement for a particular service level that applies to many but not all use cases, that we found on the STAR-TS website ( https://www.star-ts.com/about/terms-and-conditions/service-level-agreement/ ).

Along the path from Use Case to Service Level, the requester will often select an LSP to clarify the five components of the pre-production phase (see Section A). A novice requester should at least bring in an advisor to minimize risks.

A Language Services Advisor (LSA) is a new role that needs discussion, including the required skill set and how language professionals could become LSAs. Serge Gladkoff has proposed a skill set for an LSA focused on translation services. This skill set needs to be detailed but will certainly include an understanding of current AI technology, risk management, and quality management.

Note: A translation of a contract that must stand up in court needs the involvement of a professional human translator. However, there are use cases for which the risk level for raw machine translation is tolerable. We need to make such a list. Clearly, some situations, such as a 911 call that involves a human in distress, or a suicide hotline, will always require a professional human interpreter.  When is the risk level for fully automatic interpreting acceptable? (See news about the Interpreting SAFE AI task force.)

Note: A simple set of names for categories of service levels, based on risk tolerance level, is needed, for example, in the context of procurement. Service levels are primarily a consideration for providers, rather than requesters, but they are based on the requirements of the requester.

This section suggests that an important aspect of the definition of a translation service level might be to indicate whether validation of the translation output has been conducted by a professional human translator.

Validation is a term in WK46396 (the draft standard on the approach to analytic translation quality evaluation often referred to as “MQM”).  There it is applied to translation quality metrics. A valid metric measures what you are trying to measure. A metric can be part of a reliable system yet not be valid. For example, suppose a system intended to measure the weight of a person actually comes up with supposed weight based solely on height. The system can be reliable in that it comes up with the same “weight” for a person each time a given person is tested, but typically the result does not match reality. The result can be consistently wrong.

Application to Translation Service Levels

In Section A, it was proposed that a service level should be selected with the use case in mind. Otherwise, the result will probably not meet the requirements of the requester. How could we apply the technical term “validation” to service levels? One way would be to ask whether it is important to meet all the requester’s requirements. That leads to another question. Which are the most important requirements? In describing a use case, two essential components are purpose and intended audience. Why are we even translating the source text and who is the intended audience? Does it matter whether a translation meets the needs of the intended reader? If it does matter, then validation is needed, validation that the translation meets requirements, such as meeting the needs of the intended user. Validation goes beyond text.

With recent advances in Artificial Intelligence, some requesters of translation services are wondering whether all human translators will soon be replaced by machines. Others find it obvious that for at least some use cases, professional human translators will still be needed in the long term. Regardless of your position on this fundamental question, the service component “validation by a professional translator”, short for “validation of translation output by a professional human translator”, is a convenient tool to indicate whether a professional translator is checking, using human intelligence, not artificial intelligence, that all the agreed-on requirements, expressed as translation specifications, based on standardized translation parameters, have been met.

It boils down to “whom do you trust?” when the stakes are high, that is, when the consequences of a translation not meeting agreed-on specifications can be serious.

Looking at the ongoing debate about the strengths and limitations of AI, including LLMs (large language models) and systems based on technology behind LLMs (both generative AI and “neural” machine translation), it is clear that AI does not yet truly understand what it is doing. It only manipulates vast quantities of text. For now, validation is needed for some use cases.

Note that the production phase is greatly facilitated by having agreed in advance on a set of structured specifications, but the consumer is not aware of the production phase.

Consumer labels in general are a form of consumer protection or at least warning and, when applied to translation services, come into play in the postproduction phase.  They inform end-users (who “consume” text) about potential risks associated with the product in their hands. If they do not read the source language or do not have access to the source text, they cannot always detect even serious issues in the translation.

For translation, according to ASTM F2575, the output should be labeled as either UMT (Unedited Machine Translation) or BRT (Bilingually Reviewed/Revised Translation).

The first label (UMT) is used for either raw machine translation or machine translation output that has only been checked monolingually by a human. In UMT, there has been no bilingual editing by a human professional to correct Correspondence errors. Thus, UMT is a warning flag for the consumer so that they are aware there could be some “poison biscuits” somewhere in fluent text. These poison biscuits are Correspondence errors unpredictably produced by AI-based NLP (natural language processing) that would be detected by a language professional with relevant subject matter expertise.

The second label (BRT) contrasts with UMT.  It covers the spectrum from post-edited and revised machine translation to traditional human translation with no use of machine translation, so long as the entire translation has been compared bilingually by a human professional with the source, and Correspondence errors have been corrected in conformance with specifications. Of course, there could be Correspondence errors in a translation labeled BRT, but the end user is assured that a language professional has been involved in order to reduce the risk of such errors.

Thus, every translation can be labeled as either UMT or BRT. As explained above the distinction is not simply human vs machine. The UMT label is a “red flag”; the BRT label is a covenant. Ideally, the BRT label will link to the translation service provider that takes responsibility for the translation.

The concept of validation, as explained previously in this document, applies to labels. A translation labeled BRT has been validated by a competent bilingual human. A translation labeled UMT has not been validated.

Example of an SLA with a legal services provider. A basic service level for legal representation might be available only between 12:00 noon and 3:00 PM, Monday, Wednesday, and Friday and would not include meeting the client outside the office of the attorney. So, clients who are hauled off to jail, are on their own, unless they want to pay a substantial jail-visit fee. A high-end service level would include a 24-hour hotline and an agreement that a lawyer would meet the client anywhere in the city within 30 minutes of receiving a phone call. Regardless of the availability, only a lawyer can offer legal advice.

A fully-automated legal service (attention: this is a Hoosier joke) could offer something called “legal speculation” at a very low cost, where the client interacts with some kind of AI-based system.

Obviously, few would use a legal speculation service as opposed to professional legal advice. Then, why are so many using raw machine translation, which is analogous to legal speculation? Probably because the risks associated with errors in Correspondence are viewed as tolerable. Or because the requester is not even aware of the risks.

Another example: Going to a medical clinic vs using WedMD exclusively.

Point of discussion regarding ethics: Should a TSP for a particular use case where the risks are tolerable be allowed to provide raw MT (even when Google Translate does not work and a proprietary MT engine, perhaps trained on custom data, is needed)  or should they be required to refer the customer to some organization other than a TSP?