Multidimensional Quality Metrics Definition

1. Introduction

The Multidimensional Quality Metrics (MQM) provides a framework for describing and defining quality metrics used to assess the quality of translated texts and to identify specific issues in those texts. It provides a systematic framework to describe quality metrics based on the identification of textual features. This framework consists of the following items:

A vocabulary for categorizing quality issues. This vocabulary was based on an extensive examination of existing translation quality metrics. (3. Issue types (normative))
A scoring mechanism to arrive at quality scores based on either counts of errors or actual error annotations. (6. Scoring (normative)
Markup (7. Markup (normative)) including the following:
- A formal XML mechanism for declaring/describing quality metrics. This vocabulary allows implementers and users to precisely describe metrics and can be used to emulated existing metrics in MQM format. (7.1. MQM metrics description)
- A set of attributes in the mqm: namespace that can be used with XML or HTML5 (with appropriate adjustments to the HTML5 format) to embed MQM data in these file formats. These attributes are designed to work with Internationalization Tag Set 2.0 (ITS 2.0) localization quality metadata (7.2. MQM inline attributes)
- A set of elements in the mqm: namespace that can be used to insert MQM data into XML files when existing elements do not meet requirements. (7.3. MQM inline elements)
A set of guidelines for selecting issues based on the ISO/TS-11669 specification (8. Guidelines for selecting issues for use in a metric (non-normative))
A set of informative mappings from existing legacy metrics to MQM that may prove useful for implementers (9. Mappings of legacy metrics to MQM (non-normative))

MQM does not define a single metric intended for use with all translations. Instead it adopts the “functionalist” approach that quality can be defined by how well a text meets its communicative purpose. In practical terms, this statement means that

2. Terms and definitions (normative)

The following terms and definitions apply in this document.

Dimension: An aspect of a translation that defines expectations concerning the translation product. For example, “target language/locale” is the dimension that states what language/locale the translated text should appear in.
Error: An error is a specific instance of an issue that has been verified to be incorrect.
Error penalty: A numeric indication of the quantity of errors considered in determining an MQM score.
Issue: As issue is a potential problem detected in content.
Quality: Quality is the adherence of the text to specifications. In the case of translated texts, the following formulation applies:
A quality translation demonstrates required accuracy and fluency for the audience and purpose and complies with all other negotiated specifications, taking into account end-user needs.
For monolingual source texts, the formulation may be modified as follows:
A quality text demonstrates required fluency for the audience and purpose and complies with all other negotiated specifications, taking into account end-user needs.
Severity: An indication of the how severe a particular instance of an issue is. Issues with higher severity have more impact on perceived quality of the text. The default MQM severity model has three levels: minor, major, and critical.
Weight: A numerical indication of the how important a particular issue type is is overall quality assessment. The default weight for issues is 1.0. Higher numbers assign more importance to an issue type, while lower numbers assign a lower importance. A weight of 0 would indicate that an issue is checked but not counted in MQM scores. Weights serve as multipliers for error penalties in MQM scoring.

3. Principles

This section needs to be expanded and rationalized

3.1. Issues vs. errors

The term issue as used in this document refers to any potential error detected in a text, even if it is determined not to be an error. For example, if an automated process finds that a term in the source does not appear to have been translated properly, it has identified an issue. If human examination finds that the term was translated improperly, it is an error. However, examination might also find that the issue was not an error because the linguistic structure in the translation dictated that the term be replaced by a pronoun, so the translation is correct.

In most cases of translation quality assessment, issues will be errors, but with automated issue detection, some issues will not be errors. Accordingly this document refers to issues in most contexts.

For full comparability between MQM-compliant assessments, implementers should clarify whether any issues detected or scores generated are confirmed to be errors or are unconfirmed issues.

3.2. Fairness

MQM is designed to apply to (monolingual) source texts as well as translated target texts. Items described in MQM’s “Fluency”, “Design”, “Internationalization”, and “Verity” branches can apply equally well to source texts and target texts. Only the “Accuracy” branch is specific to translated texts. The default MQM scoring method allows for users to assess source texts to obtain a quality score for source texts and, if both source and target are assessed, issues found in the source may be counted against penalties for issues in the target text, resulting in higher scores. While not all implementations or usages scenarios will examine the source or count problems in the source in favor of translators, this principle is intended to help ensure that translators are recognized and credited when they have to translate inferior source texts rather than being blamed for all problems, even those beyond their control.

3.3. Flexibility

Rather than proposing a single metric for assessing all translations, MQM provides a flexible method for defining and declaring metrics that can be adapted to specific requirements. These requirements are generally stated in terms of a set of 12 “dimensions” (see Section 8.1. MQM dimensions), a subset of the translation parameters described in ISO/TS 11669:2012: Translation projects -- General guidance. Using these dimensions to define requirements and expectations before translation allows users to create appropriate metrics before translation begins and provides translators with a clear view of the criteria for assessing their work.

In addition, metrics must support both simple and sophisticated requirements. Rather than proposing yet another metric with more detail, MQM provides a flexible catalog of defined issue types that can support any level of sophistication, from a simple metric with two categories to a complex one with thirty or forty. It also supports both rubric assessment (for quick acceptance testing) and error markup/counts for cases where detailed analysis is required.

4. Conformance

Conformance of a translation quality assurance metric with MQM is determined by the following criteria:

The metric MUST use the MQM vocabulary of issue types (see Section4. Issue types (normative).)
Any issue types used by the metric that are not contained in MQM MUST be declared as MQM custom issue types at an appropriate place in the MQM hierarchy.
Custom issue types MUST NOT replace existing MQM issue types with the same semantics. (E.g., a metric that declared a “Not translated” custom issue type with the same meaning as Untranslated would not conform to MQM.)
Metrics MAY use the a custom displayName to assign a non-default name to an issue type. (E.g, a metric could declare “Not translated” to be the display name for Untranslated.) However, at least for English-language implementations, the use of the default MQM names is encouraged.
Metrics SHOULD use a declared MQM markup type (ITS 2.0-compatible) when possible for inline or standoff representation of annotated issues. However, in some instances such markup may not be possible (e.g., in non XML/HTML formats) or with tools that are used for counting errors but which do not provide inline markup.
If a quality score is calculated, it MUST conform to the MQM scoring model.
MQM-compliant metrics SHOULD generally display their issues in the MQM hierarchy, but are not required to do so (e.g., an MQM-compliant metric with six issues might opt to display them as a flat list). If issues are displayed in a hierarchy, developers are strongly encouraged, but not required, to use the MQM hierarchy. (This provision exists because legacy metrics that have their own hierarchy may be represented in MQM.)

Note that the only required aspect is use of the MQM vocabulary, which MUST NOT be contradicted or overridden.

5. Issue types (normative)

MQM defines a total of 106 issue types, as defined in this section. They are derived from an examination of major quality assessment systems, both ones based on automatic detection of issues and ones based on manual assessment by reviewers. As quality systems differ considerably in the issues they check, the MQM issue types represent a (non-strict) superset of issues found in translations (as product, as opposed to process). The superset is non-strict because it represents an abstraction of various systems and, in some cases, is less granular than actual systems. For example, an existing system might distinguish between four kinds of issues related to whitespace, but MQM would categorize all of them as Whitespace. (Users who require additional granularity or issue types not addressed in MQM should consult section 5.4. User extension on how to add user-defined categories to MQM.)

MTM issues exist in a hierarchy, with more specific issues lower in the hierarchy constituting “subtypes” of their parents. For example the issue type Terminology is a subtype of the more general issue type Accuracy. Because the issues exist in a hierarchy, rather than as a flat list, MQM can be realized at any level of granularity. At one extreme an MQM-compliant metric could check only two high-level issues, Accuracy and Fluency; at the other extreme a metric could check all 106 issues defined in MQM. In most cases the number of issues checked will be somewhere between these extremes. Guidance on selecting issue sets can be found in section 7. Creating MQM metrics. As a general rule, metrics should check the fewest number of issues possible to achieve the requirements of users.

This section first presents the hierarchy of MQM issues, followed by the detailed description of each issue type. It then contains a description of the MQM Core and mechanisms for user extension.

At the top level, MQM is defined into five major “branches”: Accuracy, Design, Fluency, Internationalization, and Verity. It also contains Other, used for issues that cannot be assigned elsewhere, and Compatibility, a branch that contains deprecated issues that are retained for compatibility with legacy systems, notably the LISA QA Model. These five main branches represet the top level in the MQM hierarchy and themselves may serve as issue types.

Issue type names are not case-sensitive (i.e., “Mistranslation”, “MISTRANSLATION”, “mistranslation”, and “MiStRaNsLaTiOn” are all equivalent). The ID values, however, are case sensitive (and are always lower-case). As a result, implementers should ensure that they do not confuse the two, even though in most cases they are nearly identical.

5.1. Hierarchical list of issue types

The following list of issue types presents the full list of MQM categories in hierarchy. Clicking on any issue type name will take the reader to the definition of the issue type in the next section.

Ordering is significant in this list. If two issue types at the same level of the hierarchy could apply, the first one in the list should be selected. For example, if it is unclear whether an issue should be categorized as Terminology or Mistranslation, Terminology should be used as it occurs first in the list. In addition, if no specific categories apply precisely at a specific level, the parent should be selected. For example, if an issue can be classified as Mistranslation but none of its daughters apply precisely, then Mistranslation should be selected.

Accuracy
- Mistranslation
  - Terminology
    - Normative terminology
  - Overly literal
  - False friend
  - Should not have been translated
  - Date/time
  - Unit conversion
  - Number
  - Entity (such as name or place)
- Omission
- Untranslated
  - Untranslated graphic
- Addition
Fluency
- Content
  - Register
    - Variants/slang
  - Style
    - Company style
    - Style guide
  - Inconsistency
    - Abbreviations
    - Images vs. text
    - Discourse
    - Terminological inconsistency
  - Duplication
  - Monolingual terminology
    - Normative monolingual terminology
  - Ambiguity
    - Unclear reference
- Mechanical
  - Spelling
    - Capitalization
    - Diacritics
  - Typography
    - Punctuation
    - Unpaired quote marks or brackets
  - Grammar
    - Word form
      - Part of speech
      - Agreement
      - Tense/mood/aspect
    - Word order
    - Function words
  - Locale violation
    - Date format
    - Time format
    - Measurement format
    - Number format
    - Quote marks type
    - National language standard
  - Character encoding
  - Nonallowed characters
  - Pattern problem
  - Sorting
  - Corpus conformance
  - Broken link/cross-reference
    - Document-internal
    - Document-external
  - Index/TOC
    - Page references
    - Index/TOC format
    - Missing/incorrect item
- Unintelligible
Verity
- Completeness
  - Lists
  - Procedures
- Legal requirements
- Locale applicability
Design
- Overall design (layout)
  - Color
  - Global font choice
  - Footnote/endnote format
  - Headers and footers
  - Margins
  - Widows/orphans
  - Page breaks
- Local formatting
  - Text alignment
  - Paragraph indentation
  - Font
  - Kerning
  - Leading
- Markup
  - Inconsistent markup
  - Misplaced markup
  - Added markup
  - Missing markup
  - Questionable markup
- Whitespace
- Graphics and tables
- Truncation/text expansion
- Length
Internationalization
Compatibility (Deprecated)
Other

These categories can be represented graphically as shown below (click on image to enlarge):

MQM graphical representation (click to enlarge)

5.2. Detailed listing of MQM issue types

This section lists all MQM issue types in alphabetical order, with the following information:

Name

The name is the English name for the issue type. This name may be localized in other languages or may be changed in a UI to reflect application-specific preferences. (For example, if an existing system is being converted to use MQM categories and already has an issue type called Terminology problem that corresponds to Terminology, the UI may display the existing name but refer to the ID value terminology internally for mapping purposes. For new English-language implementations, however, it is recommended to use the existing name to prevent confusion.)

ID	An XML identifier for the category. This ID is used to refer unambiguously to each issue type and does not change, even if a UI may display other names for the category.
Definition	A definition of the issue type
MQM Core?	(yes\|no) Specifies whether the issue is in the MQM Core or not.
Automatable?	(yes\|no) Informative: Indicates whether the issue may be automatically detected. Users interested in fully automatable subsets of MQM may wish to limit themselves to issues marked with “yes”. This specification does not provide any guidance on how to check issues automatically and detection may require language-specific modules or development. Success in detecting issues depends on factors outside the scope of this specification and individual systems may be able to identify issues not identified as automatable in this specification.
Parent	The parent of the issue type in the hierarchy. Each issue can be understood as a type of its parent.
Children	A list of any children to the current issue type.
Example(s)	One or more illustrative examples of the issue type
Note(s)	Any notes on usage for the issue type.

Jump to: [A] • [B] • [C] • [D] • [E] • [F] • [G] • [H] • [I] • [K] • [L] • [M] • [N] • [O] • [P] • [Q] • [R] • [S] • [T] • [U] • [V] • [W]

Abbreviations

ID	`inconsistent-abbreviations`
Definition	The form of abbreviations is inconsistent in the text.
MQM Core?	no
Automatable?	yes
Parent	`Inconsistency`
Children	none
Example(s)	A text uses both “app.” and “approx.” for approximately.
Note(s)

Accuracy

ID	`accuracy`
Definition	The target text does not accurately reflect the source text, allowing for any differences authorized by specifications.
MQM Core?	yes
Automatable?	no
Parent	none
Children	`Mistranslation`, `Omission`, `Untranslated`, `Addition`
Example(s)
Note(s)	Most cases of Accuracy are addressed by one of the more specific subtypes listed below. In Machine Translation literature, this category is typically referred to as “Adequacy”.

Added markup

ID	`added-markup`
Definition	The target text has markup added with no corresponding markup in the source.
MQM Core?	no
Automatable?	yes
Parent	`Markup`
Children	none
Example(s)	A source segment has no formatting tags, but the target has a set of italic tags.
Note(s)

Addition

ID
Definition	The target text includes text not present in the source.
MQM Core?	yes
Automatable?	yes
Parent	`Accuracy`
Children
Example(s)	A translation includes portions of another translation that were inadvertently pasted into the document.
Note(s)

Agreement

ID	`agreement`
Definition	Two or more words do not agree with respect to case, number, person, or other grammatical features
MQM Core?	no
Automatable?	yes
Parent	`Word form`
Children	none
Example(s)	A text reads “They was expecting a report.”
Note(s)

Ambiguity

ID	`ambiguity`
Definition	The text is ambiguous in its meaning.
MQM Core?	no
Automatable?	no
Parent	`Content`
Children	`Unclear reference`
Example(s)	A text reads “I cannot recommend this too highly.” (The meaning can be that the speaker cannot make a good recommendation or that it is highly recommended.)
Note(s)

Bold/italic

ID	`bold-italic`
Definition	Bold or italics are used incorrectly.
MQM Core?	no
Automatable?	no
Parent	`Font`
Children	none
Example(s)	A book title should have been italicized, but the italics were omitted.
Note(s)

Broken link/cross-reference

ID	`broken-link`
Definition	A link or cross reference points to an incorrect or nonexistent location
MQM Core?	no
Automatable?	yes
Parent	`Mechanical`
Children	`Document-internal`, `Document-external`
Example(s)	An HTML document has an href that points to a file that does not exist.
Note(s)

Call-outs and captions

ID	`call-outs-captions`
Definition	There are issues with call-outs (text within a graphic that identifies parts) or captions.
MQM Core?	no
Automatable?	no
Parent	`Graphics and tables`
Children	none
Example(s)	During localization the location of numbers used for call-outs has been shifted and the call-outs are no longer usable.
Note(s)

Capitalization

ID	`capitalization`
Definition	Issues related to capitalization
MQM Core?	no
Automatable?	yes
Parent	`Spelling`
Children	none
Example(s)	The name John Smith is written as “john smith”
Note(s)

Character encoding

ID	`character-encoding`
Definition	Characters are garbled due to incorrect application of an encoding.
MQM Core?	no
Automatable?	yes
Parent	`Mechanical`
Children	none
Example(s)	A text document in UTF-8 encoding is opened as ISO Latin-1, resulting in all “upper ASCII” characters being garbled.
Note(s)

Color

ID	`color`
Definition	Colors are used incorrectly
MQM Core?	no
Automatable?	no
Parent	`Overall design`
Children	none
Example(s)	Headings should be blue but are green instead.
Note(s)

Company style

ID	`company-style`
Definition	The text violates company/organization-specific style guidelines.
MQM Core?	no
Automatable?	yes
Parent	`Style`
Children	none
Example(s)	Company style states that passive sentences may not be used but the text uses passive sentences.
Note(s)

Compatibility (Deprecated)

ID	`compatibility`
Definition	The Compatibility extension contains items which may be used for compatibility with legacy metrics even though they would otherwise not be included in MQM. Most of these issue types are taken from the LISA QA Model documentation.
MQM Core?	no
Automatable?	no
Parent	none
Children	The following issue types (presented without definition) are included in the Compatibility branch: Application compatibility Bill of materials/runlist Book-building sequence Covers Deadline Delivery Does not adhere to specifications Embedded text File format Functional Output device Printing Release guide Spines Style, publishing standards Terminology, contextually inappropriate
Example(s)	A quality process checks the LISA QA Model issue “Book-building sequence” and it is included for compatibility with legacy processes
Note(s)	Use of these categories is not recommended and these issue types are to be considered deprecated. They are included only for compatibility with legacy processes. Since `Compatibility` is not a coherent category, use of this category itself is not recommended in any circumstance, although the children categories listed above may be used for compatibility.

Completeness

ID	`completeness`
Definition	The text is incomplete
MQM Core?	yes
Automatable?	no
Parent	`Verity`
Children	`Lists`, `Procedures`
Example(s)	A process description leaves out key steps needed to complete the process, resulting in an incomplete description of the process.
Note(s)	For cases where material from the source language is not present in a translation, `Omission` should be used instead.

Content

ID	`content`
Definition	Issues related to content, excluding presentational and/or mechanical issues
MQM Core?	yes
Automatable?	no
Parent	`Fluency`
Children
Example(s)	There is a problem with the presentation of information in the text
Note(s)

Corpus conformance

ID	`corpus-conformance`
Definition	The content is deemed to have a level of conformance to a reference corpus. The non-conformance type reflects the degree to which the text conforms to a reference corpus given an algorithm that combines several classes of error type to produce an aggregate rating.
MQM Core?	no
Automatable?	yes
Parent	`Mechanical`
Children	none
Example(s)	A text reading “The harbour connected which to printer is busy or configared not properly” is flagged by a language analysis tool as suspect based on its lack of conformance to an existing corpus.
Note(s)	One example of this issue type might involve output from a quality estimation system that delivers a warning that a text has a very low quality estimation score.

Date format

ID	`date-format`
Definition	A text uses a date format inappropriate for its locale.
MQM Core?	no
Automatable?	yes
Parent	`Locale violation`
Children	none
Example(s)	An English text has “2012-06-07” instead of the expected “06/07/2012.”
Note(s)

Date/time

ID	`date-time`
Definition	Dates or times do not match between source and target.
MQM Core?	no
Automatable?	yes
Parent	`Mistranslation`
Children
Example(s)	A German source text provides the date 09.02.09 (=February 9, 2009) but the English target renders it as September 2, 2009. An English source text specifies a time of "4:40 PM" but this is rendered as 04:40 (=4:40 AM) in a German translation.
Note(s)

Design

ID	`design`
Definition	There is a problem relating to design aspects (vs. linguistic aspects) of the content.
MQM Core?	no
Automatable?	no
Parent
Children	`Overall design (layout)`, `Local formatting`, `Markup`, `Whitespace`, `Graphics and tables`, `Truncation/text expansion`, `Length`
Example(s)	A document is formatted incorrectly
Note(s)	Design issues may exist either in documentions in isolation (e.g., a second-level heading is formatted as a first-level heading) or in relationship between source and target (e.g., headings are formatted differently between source and target). However, for calculation purposes, Design issues are generally included with `Fluency` issues for purposes of calculation.

Diacritics

ID	`diacritics`
Definition	Issues related to the use of diacritics
MQM Core?	no
Automatable?	yes
Parent	`Spelling`
Children	none
Example(s)	The Hungarian word bőven (using o with a double acute (˝)) is spelled as bõven, using a tilde (˜), which is not found in Hungarian.
Note(s)

Discourse

ID	`discourse`
Definition	The discourse structure of the text is inconsistent in a confusing or unclear manner.
MQM Core?	no
Automatable?	no
Parent	`Inconsistency`
Children	none
Example(s)	The text has a mixture of imperatives, descriptions of actions, and lists within a single process, making it difficult to follow the intended course of action.
Note(s)

Document-external

ID	`document-external-link`
Definition	A link or cross reference points to an incorrect or nonexistent location outside of the same document within which it occurs
MQM Core?	no
Automatable?	yes
Parent	`Broken link/cross-reference`
Children	none
Example(s)	A link in an HTML document points to a U.S. government URL that has moved and no longer exists.
Note(s)

Document-internal

ID	`document-internal-link`
Definition	A link or cross reference points to an incorrect or nonexistent location within the same document within which it occurs.
MQM Core?	no
Automatable?	yes
Parent	`Broken link/cross-reference`
Children	none
Example(s)	An internal link refers to the location “#section5” but there is no anchor “section5” in the document.
Note(s)

Duplication

ID	`duplication`
Definition	Content has been duplicated (e.g., a word or longer portion of text is repeated unintentionally).
MQM Core?	no
Automatable?	yes
Parent	`Content`
Children	none
Example(s)	A text reads “The man the man whom she saw…” A paragraph appears verbatim twice in a row.
Note(s)

Entity (such as name or place)

ID	`entity`
Definition	Names, places, or other “named entities” do not match
MQM Core?	no
Automatable?	yes
Parent	`Mistranslation`
Children
Example(s)	The source text refers to Dublin, Ohio, but the target incorrectly refers to Dublin, Ireland.
Note(s)

False friend

ID	`false-friend`
Definition	The translation has incorrectly used a word that is superficially similar to the source word.
MQM Core?	no
Automatable?	no
Parent	`Mistranslation`
Children
Example(s)	The Italian word simpatico has been translated as sympathetic in English.
Note(s)

Fluency

ID	`fluency`
Definition	Issues related to the form or content of a text, irrespective as to whether it is a translation or not.
MQM Core?	yes
Automatable?	no
Parent
Children
Example(s)	A text has errors in it that prevent it from being understood.
Note(s)	If an issue can be detected only by comparing the source and target, it MUST not be categorized as a Fluency issue.

Font, single/double-width (CJK only)

ID	`single-double-width`
Definition	Single-width characters are used when double-width are intended, or vice versa.
MQM Core?	no
Automatable?	no
Parent	`Font`
Children	none
Example(s)	A Japanese text includes カタカナ (full-width kana) when specifications required ｶﾀｶﾅ (half-width kana) instead, due to a limited display size.
Note(s)

Font

ID	`font`
Definition	Issues related to local font usage (i.e., font choices that impact a span of content rather than the global choice of the document).
MQM Core?	no
Automatable?	no
Parent	`Local formatting`
Children	`Bold/italic`, `Wrong size`, `Font, single/double-width (CJK only)`
Example(s)	Warning texts are set in sans-serif, but one of them appears in a serif font. A portion of Japanese text is set with an obliqued face (corresponding to italics in the source text) when dot accents should have been used with a non-oblique face.
Note(s)

Footnote/endnote format

ID	`footnote-format`
Definition	Footnotes or endnotes are placed inappropriately or use incorrect in-text symbols
MQM Core?	no
Automatable?	no
Parent	`Overall design`
Children	none
Example(s)	Specifications state that endnotes should be used with roman numerals but footnotes were used with in-text symbols (*, †, ‡, etc.).
Note(s)

Function words

ID	`function-words`
Definition	A function word (e.g., a preposition, “helping verb”, article, determiner) is used incorrectly.
MQM Core?	no
Automatable?	yes
Parent	`Grammar`
Children	none
Example(s)	A text reads “Check the part number as given in the screen” instead of “…on the screen”. A text reads “The graphic is then copied into an internal memory” instead of “The graphic is copied to internal memory.”
Note(s)

Global font choice

ID	`global-font-choice`
Definition	The overall font chosen is incorrect or inappropriate.
MQM Core?	no
Automatable?	no
Parent	`Overall design`
Children	none
Example(s)	A English source text uses a normal-weight serif font for body text but the Japanese translation uses a heavy-weight “gothic” (roughly, sans-serif) font appropriate for headlines only.
Note(s)

Grammar

ID
Definition	Issues related to the grammar or syntax of the text, other than spelling and orthography.
MQM Core?	yes
Automatable?	yes
Parent	`Mechanical`
Children	`Word form`, `Word order`, `Function words`
Example(s)	An English text reads “The man was in seeing the his wife.”
Note(s)

Graphics and tables

ID	`graphics-tables`
Definition	Issues related to the formatting of graphics and tables.
MQM Core?	no
Automatable?	no
Parent	`Design`
Children	`Position of graphic/table`, `Missing`, `Call-outs and captions`
Example(s)	A graphic is garbled and the wrong version is shown
Note(s)

Headers and footers

ID	`headers-footers`
Definition	Headers or footers are formatted incorrectly
MQM Core?	no
Automatable?	no
Parent	`Overall design`
Children	none
Example(s)	Headers should appear on every page but have been omitted on odd-numbered pages.
Note(s)

Images vs. text

ID	`images-vs-text`
Definition	Phrasing/wording is inconsistent between text shown in images and running text.
MQM Core?	no
Automatable?	no
Parent	`Inconsistency`
Children	none
Example(s)	A screen shot shows a button with the text “Open other…” but the text referring to the screen shot tells the user to click on the “Open alternative…” button.
Note(s)

Inconsistency

ID	`inconsistency`
Definition	The text shows internal inconsistency.
MQM Core?	yes
Automatable?	no
Parent	`Content`
Children	`Abbreviations`, `Images vs. text`, `Discourse`, `Terminological inconsistency`
Example(s)	The text states that bug reports should be submitted to a mailing list in one place and via an online bug tracker tool in another.
Note(s)

Inconsistent markup

ID	`inconsistent-markup`
Definition	Markup elements are inconsistent between the source and target
MQM Core?	no
Automatable?	no
Parent	`Markup`
Children	none
Example(s)	A target text has a set of tags for bold face in the same location where the source has tags for italics.
Note(s)

Index/TOC

ID	`index-toc`
Definition	Issues related to an index or Table of Contents (TOC).
MQM Core?	no
Automatable?	yes
Parent	`Mechanical`
Children
Example(s)	`Page references`, `Index/TOC format`, `Missing/incorrect item` A Table of Contents is missing items that should be included.
Note(s)

Index/TOC format

ID	`index-toc-format`
Definition	An index/TOC is formatted incorrectly
MQM Core?	no
Automatable?	yes
Parent	`Index/TOC`
Children	none
Example(s)	A Table of Content should be formatted with variable (hierarchical) indenting and tab leader characters, but is instead displayed as a “run-in” list.
Note(s)

Internationalization

ID	`internationalization`
Definition	There is a problem related to the internationalization of content.
MQM Core?	no
Automatable?	no
Parent	none
Children	none
Example(s)	A document assumes that all addresses use postal codes conforming to the U.S. “zip+four” convention and includes a verification step for postal codes that does not allow for non-U.S. codes.
Note(s)	As of September 2013, the intention is to expand this branch in the future with more specific issue types.

Kerning

ID	`kerning`
Definition	Kerning (inter-character spacing) is wrong.
MQM Core?	no
Automatable?	no
Parent	`Local formatting`
Children	none
Example(s)	The letters T and A in the word TAMPA are spaced too close together and collide.
Note(s)

Leading

ID	`leading`
Definition	Leading (spacing between lines of text) is off
MQM Core?	no
Automatable?	no
Parent	`Local formatting`
Children	none
Example(s)	A translated Japanese text has set lines too close together, making the text difficult to read.
Note(s)

Legal requirements

ID	`legal-requirements`
Definition	A text does not meet legal requirements as set forth in the specifications.
MQM Core?	yes
Automatable?	no
Parent	`Verity`
Children	none
Example(s)	Specifications stated that FCC regulatory notices be replaced by CE notices rather than translated, but they were translated instead, rendering the text legally problematic for use in Europe.
Note(s)

Length

ID	`length`
Definition	There is a significant discrepancy between the source and the target text lengths.
MQM Core?	no
Automatable?	yes
Parent	`Design`
Children	none
Example(s)	An English sentence is 253 characters long but its German translation is 51 characters long.
Note(s)

Lists

ID	`incomplete-lists`
Definition	A list is missing necessary items
MQM Core?	no
Automatable?	no
Parent	`Completeness`
Children	none
Example(s)	A list of items included in a retail package omits a crucial component.
Note(s)

Local formatting

ID	`local-formatting`
Definition	Issues related to local formatting (rather than to overall layout concerns)
MQM Core?	no
Automatable?	no
Parent	`Design`
Children	`Text alignment`, `Paragraph indentation`, `Font`, `Kerning`, `Leading`
Example(s)	A portion of the text displays a (non-systematic) formatting problem (e.g., a single heading is formatted incorrectly, even though other headings appear properly).
Note(s)

Locale applicability

ID	`locale-applicability`
Definition	A text does not apply to the intended locale.
MQM Core?	yes
Automatable?	no
Parent	`Verity`
Children	none
Example(s)	An advertising text translated for Sweden refers to special offers available only in Germany.
Note(s)

Locale violation

ID	`locale-violation`
Definition	The text does not adhere to locale-specific conventions or has been written for the wrong locale.
MQM Core?	yes
Automatable?	yes
Parent	`Mechanical`
Children	`Date format`, `Time format`, `Measurement format`, `Number format`, `Quote marks type`, `National language standard`
Example(s)	An incorrect format for currency is used for a German text, with a period (.) instead of a comma (,) as a thousands separator.
Note(s)

Margins

ID	`margins`
Definition	Text margins are incorrect.
MQM Core?	no
Automatable?	no
Parent	`Overall design`
Children	none
Example(s)	Specifications called for 4 cm inside margins, but 2.5 cm margins were used instead.
Note(s)

Markup

ID	`markup`
Definition	Issues related to “markup” (codes used to represent structure or formatting of text, also known as “tags”).
MQM Core?	no
Automatable?	yes
Parent	`Design`
Children	`Inconsistent markup`, `Misplaced markup`, `Added markup`, `Missing markup`, `Questionable markup`
Example(s)	Markup is used incorrectly, resulting in incorrect formatting.
Note(s)

Measurement format

ID	`measurement-format`
Definition	A text uses a measurement format inappropriate for its locale.
MQM Core?	no
Automatable?	yes
Parent	`Locale violation`
Children	none
Example(s)	A text in France uses feet and inches and Fahrenheit temperatures.
Note(s)

Mechanical

ID	`mechanical`
Definition	Issues related to the presentation and/or mechanics of the text
MQM Core?	yes
Automatable?	no
Parent	`Fluency`
Children
Example(s)	While the informational content of a text is correct, it is presented in a mechanically defective fashion.
Note(s)

Misplaced markup

ID	`misplaced-markup`
Definition	Markup is present but misplaced.
MQM Core?	no
Automatable?	yes
Parent	`Markup`
Children	none
Example(s)	A segment has three sets of paired formatting tags at the end, after the final full stop (.).
Note(s)

Missing/incorrect item

ID	`missing-incorrect-toc-item`
Definition	Items in an index/TOC are incorrect or missing
MQM Core?	no
Automatable?	yes
Parent	`Index/TOC`
Children	none
Example(s)	A chapter heading is not listed in a Table of Contents.
Note(s)

Missing graphic/table

ID	`graphics-tables-missing`
Definition	A graphic or table is missing.
MQM Core?	no
Automatable?	no
Parent	`Graphics and tables`
Children	none
Example(s)	An HTML file is missing an <img> tag, so no graphic is shown.
Note(s)

Missing markup

ID	`missing-markup`
Definition	Markup in the source is missing in the target.
MQM Core?	no
Automatable?	yes
Parent	`Markup`
Children	none
Example(s)	A source segment has a set of italic tags, but the target text does not have any tags.
Note(s)

Mistranslation

ID	`mistranslation`
Definition	The target content does not accurately represent the source content.
MQM Core?	yes
Automatable?	no
Parent	Terminology `Accuracy`
Children	`Overly literal`, `False friend`, `Should not have been translated`, `Date/time`, `Unit conversion`, `Number`, `Entity (such as name or place)`
Example(s)	A source text states that a medicine should not be administered in doses greater than 200 mg, but the translation states that it should not be administered in doses less than 200 mg.
Note(s)

Monolingual terminology

ID	`monolingual-terminology`
Definition	Terms (as opposed to general-language words) are used incorrectly.
MQM Core?	no
Automatable?	yes
Parent	`Content`
Children	`Normative monolingual terminology`
Example(s)	The term piano action should be used but piano mechanism is used instead.
Note(s)

National language standard

ID	`national-language-standard`
Definition	A text violates national language standards.
MQM Core?	no
Automatable?	yes
Parent	`Locale violation`
Children	none
Example(s)	A French advertising text uses anglicisms that are forbidden for print texts by the Academie française specifications.
Note(s)

Nonallowed characters

ID	`nonallowed-characters`
Definition	The text includes characters that are not allowed.
MQM Core?	no
Automatable?	yes
Parent	`Mechanical`
Children	none
Example(s)	A text may not include colons or forward- or back-slashes, which might cause confusion with path names on some computer systems, but it contains these characters.
Note(s)

Normative monolingual terminology

ID	`normative-monolingual-terminology`
Definition	Terms are used in violation of formal guidelines in a terminology database or other terminology resource.
MQM Core?	no
Automatable?	yes
Parent	`Monolingual terminology`
Children	none
Example(s)	A text uses the term “Acme TM200" instead of the mandated “Acme TM2000®”.
Note(s)

Normative terminology

ID	`terminology-normative`
Definition	A term is translated in a way that does not accord with its normative translation (i.e., a translation mandated in a termbase or other authoritative listing of terms and their translations that was specified for use in the translation) versus general domain usage.
MQM Core?	no
Automatable?	yes
Parent	`Terminology`
Children	none
Example(s)	A database of legal terms mandates that the English term contract be translated as Auftrag in German, but the more common Vertrag was used.
Note(s)

Number

ID	`number`
Definition	Numbers are inconsistent between source and target.
MQM Core?	no
Automatable?	yes
Parent	`Mistranslation`
Children
Example(s)	The source text specifies that a part is 124 mm long but the target text specifies that it is 147 mm long.
Note(s)

Number format

ID	`number-format`
Definition	A text uses a number format inappropriate for its locale.
MQM Core?	no
Automatable?	yes
Parent	`Locale violation`
Children	none
Example(s)	A German text has 123,456 instead of the locale-appropriate 123.456.
Note(s)

Omission

ID	`omission`
Definition	Content is missing from the translation that is present in the source.
MQM Core?	yes
Automatable?	yes
Parent	`Accuracy`
Children
Example(s)	A paragraph present in the source is missing in the translation
Note(s)

Other

ID	`other`
Definition	Used for any issues not adequately covered by the MQM core or extensions. This category should be used only if it is impossible to assign an issue to an existing category with sufficient granularity.
MQM Core?	no
Automatable?	no
Parent	none
Children	none
Example(s)	A quality process checks for errors generated from speech-to-text generated during conference interpretation. Because this error type is highly specific to the specific situation, it is not included in any predefined issue type elsewhere.
Note(s)	This category should be used only for any issue type that cannot be mapped to one of the issue types listed above. If an issue type can be considered a more granular example of an existing type, it should be categorized as that type, possibly with a custom extension if the additional granularity is needed.

Overall design (layout)

ID	`overall-design`
Definition	Issues related to overall layout and design (versus local formatting)
MQM Core?	no
Automatable?	no
Parent	`Design`
Children	`Color`, `Global font choice`, `Footnote/endnote format`, `Headers and footers`, `Margins`, `Widows/orphans`, `Page breaks`
Example(s)	A document is formatted incorrectly (e.g., it should have been set up for a print layout but instead is set up for an online presentation.
Note(s)

Overly literal

ID	`overly-literal`
Definition	The translation is overly literal.
MQM Core?	no
Automatable?	no
Parent	`Mistranslation`
Children
Example(s)	A Hungarian text contains the phrase Tele van a hocipőd?, which has been translated as “Are your snow boots full?” rather than with the idiomatic meaning of “Feeling overwhelmed?”.
Note(s)

Page breaks

ID	`page-breaks`
Definition	Page breaks appear in inappropriate locations.
MQM Core?	no
Automatable?	no
Parent	`Overall design`
Children	none
Example(s)	There is a page break between a figure and its caption.
Note(s)

Page references

ID	`page-references`
Definition	An index/TOC refers to incorrect page numbers
MQM Core?	no
Automatable?	yes
Parent	`Index/TOC`
Children	none
Example(s)	A table of contents refers to page numbers from the source document that do not apply to the translated text.
Note(s)

Paragraph indentation

ID	`paragraph-indentation`
Definition	A paragraph is indented improperly.
MQM Core?	no
Automatable?	no
Parent	`Local formatting`
Children	none
Example(s)	The first line of body paragraphs should be indented 4 mm, but some paragraphs were indented 25 mm instead.
Note(s)

Part of speech

ID	`part-of-speech`
Definition	A word is the wrong part of speech
MQM Core?	no
Automatable?	yes
Parent	`Word form`
Children	none
Example(s)	A text reads “Read these instructions careful” instead of “Read these instructions carefully.”
Note(s)

Pattern problem

ID	`pattern-problem`
Definition	The text contains a pattern (e.g., text that matches a regular expression) that is not allowed.
MQM Core?	no
Automatable?	yes
Parent	`Mechanical`
Children	none
Example(s)	The regular expression `["'”’][,\.;]` (i.e., a quote mark followed by a comma, full stop, or semicolon) is defined as not allowed for a project but a text contains the string `”,` (closing quote followed by a comma).
Note(s)

Position of graphic/table

ID	`graphics-tables-position`
Definition	A graphic or table is positioned incorrectly.
MQM Core?	no
Automatable?	no
Parent	`Graphics and tables`
Children	none
Example(s)	A text refers to Figure 1, but Figure 1 appears six pages after the point where it was referred to.
Note(s)

Procedures

ID	`incomplete-procedures`
Definition	A procedure is missing necessary steps.
MQM Core?	no
Automatable?	no
Parent	`Completeness`
Children	none
Example(s)	A document describing a procedure to restart a diesel generator omits a crucial step that must be completed prior to performing additional steps.
Note(s)

Punctuation

ID	`punctuation`
Definition	Punctuation is used incorrectly (for the locale or style)
MQM Core?	no
Automatable?	no
Parent	`Typography`
Children	none
Example(s)	An English text uses a semicolon where a comma should be used.
Note(s)

Questionable markup

ID	`questionable-markup`
Definition	Markup is present that appears malformed or inappropriate for its context.
MQM Core?	no
Automatable?	yes
Parent	`Markup`
Children	none
Example(s)	A text has opening tags but no closing tags for formatting.
Note(s)

Quote mark type

ID	`quote-mark-type`
Definition	A text uses quote marks inappropriate for its locale.
MQM Core?	no
Automatable?	yes
Parent	`Locale violation`
Children	none
Example(s)	A French text should use guillemets («») but instead systematically uses German-style quotes („”)
Note(s)

Register

ID	`register`
Definition	The text uses a linguistic register inconsistent with the specifications or general language conventions.
MQM Core?	yes
Automatable?	yes
Parent	`Content`
Children	`Variants/slang`
Example(s)	A legal notice in German uses the informal du instead of the formal Sie.
Note(s)

Should not have been translated

ID	`no-translate`
Definition	Text was translated that should have been left untranslated
MQM Core?	no
Automatable?	yes
Parent	`Mistranslation`
Children	none
Example(s)	A Japanese translation refers to “Apple Computers” as アップルコンピュータ when the English expression should have been left untranslated.
Note(s)

Sorting

ID	`sorting`
Definition	A list is not in the appropriately collated sequence.
MQM Core?	no
Automatable?	yes
Parent	`Mechanical`
Children	none
Example(s)	A listing of items should be in alphabetical order but appears in a random order instead.
Note(s)

Spelling

ID	`spelling`
Definition	Issues related to spelling of words
MQM Core?	yes
Automatable?	yes
Parent	`Mechanical`
Children	`Capitalization`, `Diacritics`
Example(s)	The German word Zustellung is spelled Zustetlugn.
Note(s)

Style

ID	`style`
Definition	The text has stylistic problems, other than those related to language register.
MQM Core?	yes
Automatable?	no
Parent	`Content`
Children	`Company style`, `Style guide`
Example(s)	A text uses a confusing style with long sentences that are difficult to understand.
Note(s)

Style guide

ID	`style-guide`
Definition	The text violates style defined in a normative style specification.
MQM Core?	no
Automatable?	yes
Parent	`Style`
Children	none
Example(s)	Specifications stated that English text was to be formatted according to the Chicago Manual of Style, but the text delivered followed the American Psychological Association style guide.
Note(s)

Tense/mood/aspect

ID	`tense-mood-aspect`
Definition	A verbal form displays the wrong tense, mood, or aspect
MQM Core?	no
Automatable?	yes
Parent	`Word form`
Children	none
Example(s)	An English text reads “After the button is pushing” (present progressive) instead of “After the button has been pushed” (past passive)
Note(s)

Terminological inconsistency

ID	`term-inconsistency`
Definition	Terminology is used in an inconsistent manner within the text.
MQM Core?	no
Automatable?	no
Parent	`Inconsistency`
Children	none
Example(s)	The text refers to a component as the brake release lever, brake disengagement lever, manual brake release, and manual disengagement.
Note(s)	This issue should not be used to cases where terminology has been translated incorrectly (Accuracy: Terminology) or cases where the wrong term is used in a source document (Fluency: Content: Monolingual Terminology).

Terminology

ID	`terminology`
Definition	A term (domain-specific word) is translated with a term other than the one expected for the domain or otherwise specified.
MQM Core?	yes
Automatable?	no
Parent	`Mistranslation`
Children	`Normative terminology`
Example(s)	A French text translates English e-mail as e-mail but terminology guidelines mandated that courriel be used. The English musicological term dog is translated (literally) into German as Hund instead of as Schnarre, as specified in a terminology database.
Note(s)

Text alignment

ID	`text-alignment`
Definition	A portion of a text is aligned inappropriately.
MQM Core?	no
Automatable?	no
Parent	`Local formatting`
Children	none
Example(s)	A heading should be left-aligned but was centered instead.
Note(s)

Time format

ID	`time-format`
Definition	A text uses a time format inappropriate for its locale.
MQM Core?	no
Automatable?	yes
Parent	`Locale violation`
Children	none
Example(s)	A text written for the U.S. uses a 24-hour time notation rather than AM/PM time.
Note(s)

Truncation/text expansion

ID	`truncation-text-expansion`
Definition	The target text has insufficient room to display the translated text according to specifications.
MQM Core?	no
Automatable?	yes
Parent	`Design`
Children	none
Example(s)	The German translation of an English string in a user interface runs off the edge of a dialogue box and cannot be read.
Note(s)

Typography

ID	`typography`
Definition	Issues related to the mechanical presentation of text. This category should be used for any typographical errors other than spelling.
MQM Core?	yes
Automatable?	yes
Parent	`Mechanical`
Children	`Punctuation`, `Typography`
Example(s)	A text uses punctuation incorrectly. A text has an extraneous hard return in the middle of a paragraph.
Note(s)

Unclear reference

ID	`unclear-reference`
Definition	The text uses relative pronouns or other referential mechanisms that are unclear as to their reference.
MQM Core?	no
Automatable?	no
Parent	`Ambiguity`
Children	none
Example(s)	A text reads “After completing this, move to the next step,” but there are a number of possible referents for this in the text.
Note(s)

Unintelligible

ID	`unintelligible`
Definition	The exact nature of the error cannot be determined. Indicates a major break down in fluency.
MQM Core?	yes
Automatable?	no
Parent	`Fluency`
Children	none
Example(s)	The following text appears in an English translation of a German automotive manual: “The brake from whe this કુતારો િસ S149235 part numbr,,."
Note(s)

Unit conversion

ID	`unit-conversion`
Definition	The target text has not converted numeric values as needed to adjust for different units (e.g., currencies, metric vs. U.S. measurement systems).
MQM Core?	no
Automatable?	yes
Parent	`Mistranslation`
Children
Example(s)	A source text specifies that an item is 25 centimeters (~10 inches) long, but the source states that it is 25 inches (63.5 cm) long.
Note(s)

Unpaired quote marks or brackets

ID	`unpaired-marks`
Definition	One of a pair of quotes or brackets—e.g., a (, ) [, ], {, or } character—is missing from text.
MQM Core?	no
Automatable?	no
Parent	`Typography`
Children	none
Example(s)	A text reads “King Ludwig of Bavaria (1845–1896 was deposed on account of his supposed madness,” omitting the closing parenthesis around the dates.
Note(s)

Untranslated

ID
Definition	Content that should have been translated has been left untranslated.
MQM Core?	yes
Automatable?	yes
Parent	`Accuracy`
Children
Example(s)	A sentence in a Japanese document translated into English is left in Japanese.
Note(s)

Untranslated graphic

ID	`untranslated-graphic`
Definition	Text in a graphic was left untranslated.
MQM Core?	no
Automatable?	yes
Parent
Children
Example(s)	Part labels in a graphic were left untranslated even though running text was translated
Note(s)

Variants/slang

ID	`variants-slang`
Definition	The text uses words such as slang that are inappropriate for the intended register.
MQM Core?	no
Automatable?	yes
Parent	`Register`
Children	none
Example(s)	A refers to dollars as “clams,” when this slang term would be inappropriate.
Note(s)

Verity

ID	`verity`
Definition	The text makes statements that contradict the world of the text
MQM Core?	yes
Automatable?	no
Parent
Children
Example(s)	The text states that a feature is present on a certain model of automobile when in fact it is not available.
Note(s)	Verity issues can apply to the source or target text and often emerge during translation when, for example, a factual statement is tru in the source locale but not true in the target locale.

Whitespace

ID	`whitespace`
Definition	Whitespace is used incorrectly
MQM Core?	no
Automatable?	yes
Parent	`Design`
Children	none
Example(s)	A document uses a string of space characters instead of tabs Extra spaces are added at the start of a string
Note(s)

Widows/orphans

ID	`widows-orphans`
Definition	The text has widows or orphans (single or short lines of text that appear on a separate page from the rest of a paragraph).
MQM Core?	no
Automatable?	no
Parent	`Overall design`
Children	none
Example(s)	Specifications state that at least two lines of a paragraph must appear on a page (if the paragraph is more than one line), but a single line starts a page while two appear on the previous page.
Note(s)

Word form

ID	`word-form`
Definition	There is a problem in the form of a word
MQM Core?	no
Automatable?	yes
Parent	`Grammar`
Children	`Part of speech`, `Agreement`, `Tense/mood/aspect`
Example(s)	An English text has becomed instead of became.
Note(s)

Word order

ID	`word-order`
Definition	The word order is incorrect
MQM Core?	no
Automatable?	yes
Parent	`Grammar`
Children	none
Example(s)	A German text reads “Er hat gesehen den Mann” instead of “Er hat den Mann gesehen.”
Note(s)

Wrong size

ID	`wrong-font-size`
Definition	The font size is incorrect
MQM Core?	no
Automatable?	no
Parent	`Font`
Children	none
Example(s)	A legal notice should be set in a 9 pt size, but was instead set in 7 pt.
Note(s)

5.3. MQM Core

In order to simplify the application of MQM, MQM defines a smaller “Core” consisting of 21 issues types that represent the most common issues arising in linguistic quality assurance of translated texts. The Core does not address formatting and many applications may wish to add items from the Format branch to the Core. The Core represents a relatively high level of granularity suitable for many tasks. Where possible, users of MQM are encouraged to use issues from the Core to promote greater interoperability between systems.

The 21 issues defined in the MQM core are as follows:

Accuracy
- Mistranslation
  - Terminology
- Omission
- Addition
- Untranslated
Verity
- Completeness
- Legal requirements
- Locale applicability
Fluency
- Content
  - Register
  - Style
  - Inconsistency
- Mechanical
  - Spelling
  - Typography
  - Grammar
  - Locale violation
- Unintelligible

Even the 21 issues of the Core represent more issues than are likely to be checked in any given quality application and users may define subsets of the core for their needs. It is recommended for translation quality assessment tasks that issues contain at least the issue types Accuracy and Fluency.

5.4. User extension

While users are strongly encouraged to limit issue types to pre-defined MQM issues, they may add additional issue types to MQM to meet additional requirements. User-defined issue types MUST include the following information:

Name: A human-readable name for the issue type.
ID: An unprefixed QName that serves as the XML identified of the issue type. The ID MUST begin with x- to indicate that the issue is a user-defined issue. E.g., x-respeaking-error would be a valid ID for a “respeaking error” but respeaking-error would be invalid.
Parent: The ID value of the parent issue type. The parent may by a predefined MQM issue type or a user-defined issue type.
Definition: A human-readable explanation of the issue type.

User extensions do not provide interoperability between systems and impede the exchange of data. Nevertheless they may be needed to support requirements not anticipated in MQM. Users should tie extensions into the predefined hierarchy using the parent value as much as possible since doing so provides consumers of MQM data with the best guidance in interpreting unknown categories and mapping them to other systems. As with other aspects of MQM, users should limit granularity to the least granular level that meets requirements.

Users who encounter frequent need for custom extensions are encouraged to communicate their requirements to the MQM project for possible inclusion of these types in future versions of MQM.

6. Scoring (normative)

The MQM scoring model applies only to error-count implementations of MQM. This specification does not define a scoring model for rubric systems, which are more variable in nature than error-count metrics.

MQM can generate document quality scores according to the following formula:

TQ = 100 - AP - ({FP}_{T} - {FP}_{S}) - ({VP}_{T} - {VP}_{S})

where:

$TQ$ = quality score: The overall rating of quality
$AP$ = penalties for Accuracy: Sum of all weighted penalty points assigned in the Accuracy branch
$FPT$ = Fluency penalties for the target: Sum of all weighted penalty points in the target text assigned to the Fluency branch. (Note: for computational purposes, Design and Internationalization are treated with Fluency.)
$FPS$ = Fluency penalties for the source: Sum of all weighted penalty points in the source text assigned to the Fluency branch. If the source is not assessed FPS = 0.
$VPT$ = Verity penalties for the target: Sum of all weighted penalty points in the target text assigned to the Verity branch
$VPS$ = Verity penalties for the source: Sum of all weighted penalty points in the source text assigned to the Verity branch. If the source is not assessed VPS = 0.

All penalties are relative to the sample size (in words) and are calculated as follows (assuming default weights and severity levels):

P = \frac{({Issues}_{minor} + {Issues}_{major} \times 5 {Issues}_{critical} \times 10)}{Word count}

where:

${Issues}_{minor}$	= Number of issues with a “minor” severity
${Issues}_{major}$	= Number of issues with a “major” severity
${Issues}_{critical}$	= Number of issues with a “critical” severity

A score can thus be generated through the following (pseudo-code) algorithm:

foreach accuracyIssue {
	accuracyIssueTotal = accuracyIssueTotal +
	(accuracyIssue * weight[accuracyIssueType] * severityMultiplier);
}

foreach targetFluencyIssue {
	targetFluencyIssueTotal = targetFluencyIssueTotal +
	(targetFluencyIssue * weight[targetFluencyIssueType] * severityMultiplier);
}

foreach sourceFluencyIssue {
	sourceFluencyIssueTotal = sourceFluencyIssueTotal +
	(sourceFluencyIssue * weight[sourceFluencyissueType] * severityMultiplier);
}

foreach targetVerityIssue {
	targetVerityIssueTotal = targetVerityIssueTotal +
	(targetVerityIssue * weight[targetVerityIssueType] * severityMultiplier);
}

foreach sourceVerityIssue {
	sourceVerityIssueTotal = sourceVerityIssueTotal +
	(sourceVerityIssue * weight[sourceVerityIssueType] * severityMultiplier);
}

// Generate subscores
accuracySubScore = accuracyIssueTotal / wordCount;
fluencySubScore = (targetFluencyIssueTotal - sourceFluencyIssueTotal) / wordCount;
veritySubScore = (targetVerityIssueTotal - sourceVerityIssueTotal) / wordCount;

// Generate overall score
translationQualityScore = 100 - accuracySubScore - fluencySubScore - veritySubScore;

In this algorithm, each issue type has a weight assigned by the metric that is retrieved and used to determine the individual penalties. Penalties are cumulative. Note that if the source is examined, penalties against the source are effectively added to the overall score for the translation, reflecting the fact that they indicate problems in the source the translator had to deal with. If the source is not assessed, the source penalties are by definition 0 and do not count for or against the translation’s quality score.

Default values for error-count metrics

For the purposes of calculating quality scores, the following default values apply:

Weight

All issues have a default weight of 1.0. This weight can be updated on a per-issue basis to reflect specific requirements.

Severity

The default severity levels are defined as follows:

minor: 1.0. Minor issues are issues that do not impact usability or understandability of the content. For example, if an extra space appears after a full stop, this may be considered an error, but does not render the text difficult to use or problematic (even if it should be corrected).
major: 5.0. Major issues are issues that impact usability or understandability of the content but which do not render it unusable. For example, a misspelled word may require extra effort for the reader to understand the intended meaning, but do not make it impossible.
critical: 10.0. Critical issues are issues that render the content portion unfit for use. For example, a particularly bad grammatical error that changes the meaning of the text would be considered critical.

The default severity weights are taken from the LISA QA Model and represent common industry practice. This practice has not been experimentally validated and other values may be more appropriate for specific purposes (e.g., for an important legal text it might be appropriate to assign a higher value to critical such that even a single critical error will cause the text to fail). Users MUST declare any values that differ from the default values.

7. Markup (normative)

This section describes the MQM declarative markup. Use of the metrics declaration markup is mandatory for declaring an interoperable MQM metric. When used with XML or HTML, it is strongly recommended that the ITS 2.0 Localization Quality Issue data category be used to declare MQM issues in conjunction with the locQualityProfileRef pointing to a valid MQM definition. Note that when implemented with ITS 2.0 quality markup that the requirements for implementing are also mandatory.

7.1. MQM metrics description

MQM provides an XML mechanism for exchanging descriptions of MQM-compliant metrics. MQM metrics description files use the .mqm file name extension. An .mqm file contains a hierarchical list of MQM issue types. This listing MUST conform to the hierarchy of issue types.

The following is an example of a small metric description file with issue names in both English and German. It includes a user-defined extension (x-respeaking) used to identify errors cause by speech recognition when a vocal text read from a text stream is misinterpreted.

<?xml version="1.0" encoding="UTF-8"?>
<mqm version="2.0">
  <head>
    <name>Small metric</name>
    <descrip>A small metric intended for human consumption</descrip>
    <version>1.5</version>
    <src>http://www.example.com/example.mqm</src>
  </head>
  <issues>
    <issue type="accuracy" display="no">
      <issue type="mistranslation" display="no">
        <issue type="terminology" weight="1.5"/>
      </issue>
      <issue type="omission" weight="0.7"/>
      <issue type="addition"/>
    </issue>
    <issue type="fluency" display="no">
      <issue type="content" display="no">
        <issue type="style" weight="0.5"/>
      </issue>
      <issue type="mechanical" display="no">
        <issue type="spelling"/>
        <issue type="grammar"/>
      </issue>
      <issue type="unintelligible" weight="1.5"/>
    </issue>
    <issue type="x-respeaking" weight="1.5"/>
  </issues>
  <displayNames>
    <displaNameSet lang="en">
      <displayName typeRef="accuracy">Adequacy</displayName>
      <displayName typeRef="terminology">Terminology</displayName>
      <displayName typeRef="omission">Omission</displayName>
      <displayName typeRef="addition">Addition</displayName>
      <displayName typeRef="fluency">Fluency</displayName>
      <displayName typeRef="content">Content</displayName>
      <displayName typeRef="style">Style</displayName>
      <displayName typeRef="mechanical">Mechanical</displayName>
      <displayName typeRef="spelling">Spelling</displayName>
      <displayName typeRef="grammar">Grammar</displayName>
      <displayName typeRef="unintelligible">Unintelligible</displayName>
      <displayName typeRef="x-respeaking">Respeaking</displayName>
    </displaNameSet>
    <displayNameSet lang="de">
      <displayName typeRef="accuracy">Genauigkeit</displayName>
      <displayName typeRef="terminology">Terminologie</displayName>
      <displayName typeRef="omission">Auslassung</displayName>
      <displayName typeRef="addition">Ergänzung</displayName>
      <displayName typeRef="fluency">Sprachkompetenz</displayName>
      <displayName typeRef="content">Inhalt</displayName>
      <displayName typeRef="style">Stil</displayName>
      <displayName typeRef="mechanical">Mechanisch</displayName>
      <displayName typeRef="spelling">Rechtschreibung</displayName>
      <displayName typeRef="grammar">Grammatik</displayName>
      <displayName typeRef="unintelligible">Unverständlich</displayName>
      <displayName typeRef="x-respeaking">Sprecherfehler</displayName>
    </displayNameSet>
  </displayNames>
  <severities>
    <severity name="minor" multiplier="1"/>
    <severity name="major" multiplier="5"/>
    <severity name="critical" multiplier="10"/>
  </severities>
</mqm>

7.2. MQM inline attributes

MQM implements the following attributes in the mqm namespace:

issueType. Contains the MQM issue type, listed by ID. Note: MQM implementations MUST use the ID and MUST NOT use a localized name.
issueSeverity. Contains the MQM issue severity using the name defined in the metric. Note that default values are minor, major, and critical. While other values MAY be used, if they are used they MUST be defined in the metric definition for proper interpretation.

MQM is designed to be used in conjunction with the following ITS 2.0 attributes from the localization quality issue data category:

locQualityIssueType: Contains the issue type as defined by ITS 2.0. Mapping the native MQM value to the appropriate ITS issue type helps ensure compatibility with ITS 2.0-aware implementations, even if they do not implement MQM.
locQualityIssueComment: Contains a human-readable comment about the issue.
locQualityIssueSeverity: Contains a rating of severity from 0 to 100. Mapping from the name contained in the MQM issueSeverity attribute to this attribute enables ITS 2.0-aware tools to interpret the severity of the issue.

To ensure compatibility with ITS 2.0 markup, implementers SHOULD use ITS 2.0 markup where possible. All of the ITS 2.0 localization quality annotation may be used. MQM markup adds capability to the ITS 2.0 quality markup.

			
<?xml version="1.0"?>
<doc xmlns:its="http://www.w3.org/2005/11/its" its:version="2.0">
<doc xmlns:mqm="[XXXXXXXXXXX]" mqm:version="1.0">
  <para><span 
      mqm:issueType="spelling"
      mqm:issueSeverity="major"
      its:locQualityIssueType="misspelling"
      its:locQualityIssueComment="Should be Roquefort"
      its:locQualityIssueSeverity="50">Roqfort</span> is an cheese</para>
</doc>

To create this markup the following process is followed:

The MQM issue type (spelling) is mapped to the corresponding ITS 2.0 type (ITS 2.0 is less fine-grained than MQM in many cases) and added as the value of its:locQualityIssueType.
The MQM issue type and severity are declared in the mqm: namespace
The value of the severity multiplier is declared on a scale from 0 to 100 and inserted as the value of the its:locQualityIssueSeverity attribute. In this case the multiplier value was 5 (out of 10), so it is represented as 50 in ITS markup.
A comment is added using the its:locQualityIssueComment attribute.

7.3. MQM inline elements

In general, MQM XML implementations should use existing span-level elements in the native XML format that MQM is being added to where possible. This use can be done using any of the ITS 2.0 methods with the addition of the MQM-specific attributes. However, such elements may not be available. In such cases, MQM defines two elements that can be used to add inline markup:

<mqm:startIssue />. This element defines the starting position of an MQM span.
<mqm:endIssue />. This element defines the end position of an MQM span.

Two empty elements are used so as to prevent any interference between MQM tags and existing XML structure, such as those that could be caused by improperly nested elements. To pair these tags the id attribute is used. ID values MUST be unique within the document to prevent confusion.

An example of an MQM annotation is seen in the following XML snippet:

    <para>“Instead of strengthening
        <mqm:startIssue type="function-words" id="1f59a2" severity="minor" agent="f-deluz" comment="article unneeded here" active="yes">the<mqm:endIssue id="1f59a2">
        civil society, the president cancels
        <mqm:startIssue type="agreement" severity="major" comment="should be “it”" agent="f-deluz" id="3c469d" active="yes"/>them<mqm:endIssue id="3c469d"/>
        de facto”, deplores Saeda.
    </para>

The mqm:startIssue element MUST take the following mandatory attributes:

id. Used to match the corresponding mqm:startIssue and mqm:endIssue tags within text
type. Provides the MQM issue type.

The mqm:startIssue element CAN take the following optional attributes:

severity. Permissible values defined by the MQM metric in use. Provides the severity of the issue. Default value is undefined.
agent. Text string identifying the agent that supplied the annotation. Default value is undefined.
comment. Text string containing a human-readable comment attached to an issue. Default value is undefined
active. One of yes or no. Indicates whether the issue is considered active (yes) or inactive (no). Default value is yes. If an issue is marked as inactive, this means that it has either been resolved or determined to be in error.

In addition, if ITS 2.0 attributes can be added to these elements, if their use is desirable.

The mqm:endIssue element MUST take the following mandatory attribute:

id. The id value corresponding to the mqm:startIssue tag that begins the identified span.

Use of these inline elements also requires that the mqm namespace be declared in the document. The method for declaring this namespace needs to be determined.

8. Creating MQM metrics (non-normative)

8.1. MQM dimensions

MQM makes use of a selection of 11 of the 21 parameters defined in ISO/TS-11669, with the addition of one additional parameter, Output modality, not directly addressed in the current version of ISO/TS-11669. In MQM, these 12 items are termed “Dimensions”. The dimensions are defined as follows:

Dimension	Description
1. Language/locale	Definition: The language into which the text is to be translated Note/Explanation: This parameter should specify geographical language variants where appropriate. Examples: the text is to be translated into Swiss German (de-CH) the text is to be translated into Cantonese as spoken in Hong Kong using Traditional Chinese characters (zh-HK-Hant)
2. Subject field/domain	Definition: Subject field(s) (domain(s)) of the source text Note/Explanation: This information should be as specific as possible to assist translation providers in finding the best translators for the job Examples: the text is a specialized text dealing with meteorological science the text is a sixteenth-century legal text regarding fishing rights in the North Sea
3. Teminology	Definition: List of terms or reference to terms to be used Note/Explanation: These terms are domain- or project-specific ones Examples: the requester provides instructions to see a website that defines many of the domain-specific terms in the project the requester states that specialist physics terms are to be used
4. Text type	Definition: The type and genre of the source content Note/Explanation: Needed to locate resources with the appropriate linguistic skills. For example, a translator who specializes in technical translations may not be ideal to translate a compilation of 12th-century religious poems. Examples: user manual literary novel set in medieval Ireland
5. Audience	Definition: The project’s target audience Note/Explanation: The audience should be described or defined as precisely as possible without being too restrictive Examples: business analysts with a background in Russian mineral exploration activity teenage users of tablet computers
6. Purpose	Definition: statement of the purpose or intended use of the translation Note/Explanation: This information is useful in helping the translator decide the appropriate manner in which to translate the text. In some cases the purpose of the translation may differ significantly from the purpose of the source text. Examples: the text is intended for entertainment, to transmit information, or to persuade an audience of a political point the source text was written to convince youth to join a political movement but the translation is to used by foreign journalists to help them understand the goals of this political movement
7. Register	Definition: Description of the linguistic register to be used in the target language Note/Explanation: Register is often difficult to infer from the source text and must be defined on a per-language basis Examples: the text is an informal conversation between friends and should be translated in German using the du form the text is a formal letter to the Hungarian ambassador and should be translated using the Őn pronouns and very formal honorifics, salutations, and grammatical structures
8. Style	Definition: Information about the document’s style. Note/Explanation: Could include formal style guides, references to comparable documents, or other clear indications of style expectations Examples: the text is a promotional piece for investors and style is highly relevant, with the translation trying to capture an air of excitement the text is intended for use by technicians in a service environment and style is considered irrelevant the text is to be published by a press with very specific in-house style rules that must be followed
9. content correspondence	Definition: Specifies how the content is to be translated Note/Explanation: The default assumption is that text is to be fully translated and adapted to the target locale (a covert, localized translation). In some instances, requesters may ask for partial or summary translations Examples: a British English text should be fully translated into German but all prices should be left in pounds sterling rather than converted to euros a marketing text should be heavily adapted to match target language conventions, with the translator free to rewrite portions as needed to appeal to the audience the text should be translated as a summary that presents the main points but leaves out details
10. Output modality	Definition: Information about the way in which the translated text will be displayed/presented Note/Explanation: This dimension provides information about the specific environments in which the text will be output and any limitations or special requirements they may impose. Examples: the text is to be output as captions on a YouTube video the text will be used in voice prompts for a telephone dialogue system with a female voice reading the prompts the text will be displayed on an embedded LCD screen of a device and is limited to a length of 25 characters
11. File format	Definition: The file format(s) in which the translated content is to be delivered Note/Explanation: It is quite common for the target file format to differ from the source file format Examples: the translator is asked to translate a text in an InDesign file but to return the translation as an RTF text the translator is to return text in Microsoft Word (.docx) format and graphics in layered TIFF format
12. Production technology	Definition: Any technology or software to be used in the translation process Note/Explanation: May be generic or specific as to particular translation tools Examples: the project is to be completed using a translation memory tool of the translator’s choice the translation must use TTC TermBase v3

After the values for these dimensions are fully specified, MQM implementers should verify that the selection of issue types will ensure that the requirements defined by the dimensions are met. Note that dimensions may override each other. For example, under Content correspondence the dimensions might specify that a “gist” translation is acceptable, in which case Style would not normally be assessed; however if Audience specifies that the target audience consists of young readers with low literacy, Style might be assessed to assure that the “simple” style needed for the target audience is achieved.

At this stage in MQM development, there are no normative guidelines for selecting issues. Instead implementers are encouraged to go through each dimension to identify project-relevant issues that will enable them to verify whether the translation meets the requirements set out in those dimensions. Future versions of MQM may provide a more formal approach to issue selection.

8.2. Error-count metrics

Error-count metrics are created by making a selection of relevant issues from the listing of MQM issue types. The following procedure may be used to create a metric:

Complete a full set of project specifications, including the 12 MQM dimensions. Ensure that all stakeholders are in agreement about the values of the parameters. (Note that the value of some dimensions, such as the target language, may change from project to project, so implementers should consider the range of likely values. For example, if a project will be translated into 15 languages, the impact each language might have should be considered.)
For the value of each dimension, consider what features of the text would be needed to verify that the text meets specifications and note these issue types down. Note that “doesn’t matter” is an acceptable value for many dimensions and if this value is chosen, the dimension may be skipped. (E.g., if Style is judged to be insignificant, then this dimension will be skipped in assessment.
After deciding what features need to be checked, determine which issue types can be used to assess that feature and note these types.
From the list of issue types, prioritize them based on the importance of each dimension and then make a selection of issue types based on this list and the priorities. (Note that it may be impractical to do fine-grained analysis of every potential issue type identified. Feedback from LSPs suggests that six to seven issue types is sufficient for most assessment tasks, although some use up to twenty.
If a score is to be assigned, assign weights to the issues. Assigning weights is a tricky process and should be done by assessing existing translations deemed to be acceptable, borderline acceptable, and unacceptable to see what impact each issue type has on that judgment. Note that some existing metrics, such as SAE J2450, have predefined weights that should be honored. The default issue weight in MQM is 1.0 and any positive decimal value may be used.
If the resulting metric is to be implemented in an MQM-compliant tool chain, it should be declared as described in Section 7.1. MQM metrics description.

When considering which issues to check, creators of metrics should consider the following practical guidelines:

Are there any requirements for compatibility with legacy systems or standard/semi-standard specifications? If so, choose issue types that correspond to those used by those systems/specifications. In most cases it is possible to emulate legacy metrics in MQM with little or no modification, although some might require the use of custom extensions.
Select the least granular issue types that allow assessment of whether the text meets specifications. For example, in many cases use of the category Grammar would be sufficient because it is not particularly relevant to know what subcategory is used. On the other hand, when trying to diagnose problems generated by an MT system, finer-grained types might be necessary.
When possible, choose issues from the MQM Core. Using these issues helps ensure compatibility. However, the Core does not cover all cases, including common ones such as checking formatting, because it is focused on text translations.
Consider not just requirements for one set of specifications/dimensions, but also for other likely sets. For example, if two types of translations are frequently assessed, it may make sense to develop one list of issues with different sets of weights and to use the single (master) set of issues. This practice is recommended to prevent the need to train evaluators on multiple metrics.

8.3. Rubric metrics

Rubric assessment methods are more flexible in some respects than error-count metrics. They are designed to provide an assessment of the translated text as a whole rather than a detailed accounting of all errors. As error-count (analytic) assessment can be time consuming and is not needed in all cases (e.g., when the question is whether a text should be accepted or not), rubric methods may be more appropriate in some cases. Although the primary focus of the MQM tools is on error-count analysis, most of the MQM issue types can be easily converted into rubric types that apply to the text as a whole. For example, the MQM Punctuation issue type can be used by asking assessors using a rubric tool whether the text is punctuated correctly. In this context some issues will be more useful than others. For example, the Pattern problem issue type is unlikely to be useful in most rubric assessments since it generally makes sense only with regard to very specific sections of a text. By contrast, categories like Grammar can more readily be applied to entire texts.

Note that there is no single method for building rubric scores. In a rubric approach specific issues are addressed through qualitative questions that may be assessed on a binary- or scalar-value system. For example, a rubric assessment might address the Spelling issue via questions like the following:

The translated text is spelled correctly:
[ ] Yes
[ ] No
The translated text is spelled correctly:
[ ] Strongly disagree
[ ] Disagree
[ ] Neither disagree nor agree
[ ] Agree
[ ] Strongly agree
Does the translated text meet expectations with regard to correct spelling?
[ ] The text does not meet expectations
[ ] The text meets expectations
[ ] The text exceeds expectations

Because the scoring for rubric systems is highly dependent on the type of assessment scale used, no specific scoring system is provided here. Users of MQM who wish to implement it in a rubric environment should tie rubric questions to specific MQM issue types and develop appropriate scoring systems. This version of MQM does not define a system for describing rubric scoring systems, although future versions may do so. However, by using the MQM issue types and associating specific rubric questions with them, implementers can make their metrics more transparent and tie them to project dimensions in the same way that can be done with error-count metrics.

The following guidelines may assist in designing appropriate rubric assessments and selecting issue types:

Use the highest-level issues that suffice to provide the needed assessment. For example, “Is the text translated accurately?” (assessing Mistranslation) is more likely to be answered accurately than “Does the text show any problems related to ‘false friends’?” (assessing False friends), especially as false friends are less likely to occur than generalized mistranslation. However, for some assessment purposes, fine-grained rubrics may be appropriate.
Write rubric statements or questions that clearly address the desired issue type. For example, “Does the text comply with specified guidelines for style?” is more likely to obtain an accurate assessment for Style than “Does the text read well?”, which is vague in its intent.
Design rubric questions that allow assessors to give credit, not just penalties. Rubric assessment tools often allow assessors to indicate that a text outperforms expectations in order to give credit for jobs well done. Using scalars that give translators credit is in line with the MQM principle of fairness and is roughly analogous to the MQM practice of assigning credits to translators for errors detected in the source.

9. Mappings of legacy metrics to MQM (non-normative)

To be added

10. Acknowledgements

Portions of this document were developed as part of the Coordination and Support Action “Preparation and Launch of a Large-scale Action for Quality Translation Technology (QTLaunchPad)”, funded by the 7th Framework Programme of the European Commission through the contract 296347.

11. Previous versions (non-normative)

Version 0.1.5 (2014-02-14)
Version 0.1 (2014-02-11) - First draft

Changes since last version

Added error-count description
Added mqm inline elements description
Added rubric description
Added list of dimensions and description of dimensions
Various corrections
Added copyright notice and acknowledgement of QTLaunchPad

Changes from version 0.1 to 0.1.5

Added MQM graphic representation.

Multidimensional Quality Metrics (MQM) Specification

Copyright

Editor

Contributors

Document status

Feedback

Overview

Table of Contents