Photo of Terence Eden. He has a beard and is smiling.

EBCDIC is incompatible with GDPR

By · · 31 comments · 400 words · read ~38,728 times.

Welcome to acronym city!

The Court of Appeal of Brussels has made an interesting ruling. A customer complained that their bank was spelling the customer's name incorrectly. The bank didn't have support for diacritical marks. Things like á, è, ô, ü, ç etc. Those accents are common in many languages. So it was a little surprising that the bank didn't support them.

The bank refused to spell their customer's name correctly, so the customer raised a GDPR complaint under Article 16.

The data subject shall have the right to obtain from the controller without undue delay the rectification of inaccurate personal data concerning him or her.

Cue much legal back and forth. The bank argued that they simply couldn't support diacritics due to their technology stack. Here's their argument (in Dutch - my translation follows)

Dutch text and a diagram.

Bank X also explained that the current customer data management application was launched in 1995 and is still running on a US manufactured mainframe system.
This system only supported EBCDIC ("extended binary-coded decimal interchange code"). This is an 8-bit standard for storing letters and punctuation marks, developed in 1963-1964 by IBM for their mainframes and AS/400 computers. The code comes from of the use of punch cards and only contains the following characters…

(Emphasis added.)

EBCDIC is an ancient (and much hated) "standard" which should have been fired into the sun a long time ago. It baffles me that it was still being used in 1995 - let alone today.

Look, I'm not a lawyer (sorry mum!) so I've no idea whether this sort of ruling has any impact outside of this specific case. But, a decade after the seminal Falsehoods Programmers Believe About Names essay - we shouldn't tolerate these sorts of flaws.

Unicode - encoded as UTF-8 - just works. Yes, I'm sure there are some edge-cases. But if you can't properly store human names in their native language, you're opening yourself up to a lawsuit.

Source

GDPRhub - 2019/AR/1006

Dance

Reactions


https://twitter.com/joachimesque/status/1450746564100730882

Share this post on…

31 thoughts on “EBCDIC is incompatible with GDPR”

    1. Lee Willy Minifees says:

      As far as I know, most countries have laws that regulate what you can put into your legal name.

      Reply

  4. says:

    EBCDIC has many code pages, just like DOS, and by selecting the correct one you can encode characters from any European language you want. So the bank's argument is not completely correct.

    Reply

    1. Dror Harari says:

      Correct but given that there are EBCDIC code page for every country which are not consistent (even the encoding of simple characters like $ may change from one country to another), this prevents a central application from supporting multiple code pages (sets of characters). You would need to store, along with the name, the code page that is used and then add program code to deal with that, something that is not practical.

      Suing for this under GDPR makes zero sense. If your bank is an ancient dinosaur, switch bank.

      Reply

      1. @edent says:

        "If you're being discriminated against, just take some time, money, and effort go where you won't be."

        How about "Not supporting a diverse range of customers doesn't make sense. If you can't do that, shut down your organisation."?

        Reply

  6. Jan says:

    I‘m happy. It feels like revenge served very cold. I tried to open a Barclays account in 2006 and have a German last name with an ö. The Lady at the bank said she had to spell the name exactly as on my id. I said, use an ö. She said I don’t have one on my keyboard. I said then use oe instead. She said she couldn’t, because she had to spell it exactly like it was on my id. And on and on.

    Reply

    1. JuggleT says:

      if it is a german id just show the machine readable part there the name is written with ae, oe, ue or ss

      Reply

      1. Jan says:

        Didn’t know, thanks! It’s 15 years ago, so I doubt Incan still find her…

        Reply

      2. Erkin Alp Güney says:

        Same in Turkish IDs. Machine readable portion spells my last name as Gueney.

        Reply

  7. says:

    Wow. This is mad bonkers, and I shall be raising this with several places that can’t get my name right forthwith!

    Reply

  8. Jan (2) says:

    "Unicode - encoded as UTF-8 - just works. Yes, I'm sure there are some edge-cases. But if you can't properly store human names in their native language, you're opening yourself up to a lawsuit."

    Those edge cases are for a large part in human names. There are rare Chinese characters that are not in unicode, those are rare because they are only used in a few names. And one can question if a language like Chinese with a long tail of very rare characters is not effectively an open-ended set. Someone invented those characters in the past, so why won't that process continue?

    All of that is not really relevant to the legal question as judges tend to take into account what is reasonable in the current day and age, which according to this court is to support at least accents.

    Reply

    1. Erkin Alp Güney says:

      In Chinese, you could at least use a combining backspace to split characters into two existing ones and thus overtype two characters together.

      Reply

  9. says:

    This is not a technical limitation — come up with an encoding just like UTF-8. Encode where possible in EBCDIC, but choose a bit to indicate higher chunks are available. Migrate legacy data to the new encoding, keeping an eye out for corner cases. Tricky bit is that these old bank systems tend to have fixed-width fields, which can mess with multi-byte encodings. I did something like this back when I worked on compilers for IBM as a work-around for our test suites sometimes having utf-8 filenames. Fairly easy to make a idempotent transformation. I should have gone full into consulting! sheesh.

    Reply

  10. Karl Williamson says:

    UTF-EBCDIC allows encoding all Unicode code points, similarly to UTF-8.
    https://www.unicode.org/reports/tr16/tr16-8.html
    There are modern Perl 5 releases available that support this which I run on z/OS; Python also is claimed to support EBCDIC, but I don't have experience with it regarding Unicode.

    Both EBCDIC 1047 and 037 code pages are isomorphic to Latin1. Almost all European languages should be directly encodable via these.

    Reply

  16. Dave Cridland says:

    The bank could just use punycode in EBDIC of course. Just try saying that out loud author throwing up a bit.

    Reply

  18. Blair Wyman says:

    A point worth mentioning, IMHO, is that this banking application
    was apparently designed and written in the 1990's, and has been
    serving its intended purpose for almost 30 years.

    If the Y2K or Euro character events did not break it --
    and I have no reason to suspect that -- this application
    may theoretically be unchanged since the day it was written.

    Is that a Good Thing? ...or a Bad Thing? I dunno.
    I just know it is a Thing.
    it is a Thing.

    Reply

  19. Timothy says:

    Jim Rees and others are correct, and the headline is incorrect. EBCDIC isn't the culprit. EBCDIC has had codepages for eons, and that'd be one classic way the bank could solve this problem -- or should have solved this problem decades ago. It's a well solved problem. Another way, probably better nowadays, is to use Unicode (UTF-8 probably). Whether it's IBM Z or IBM i, these systems definitely support Unicode and have since the 1990s. The implementation could be in hybrid-quick-hacky fashion. For example, put some "trigger/escape code" in the existing name field (with the current not great EBCDIC codepage choice) that then points to a UTF-8 encoded name stored alongside. It'd require an application code change, sure, but it's not rocket science actually.

    Here's the real headline: "Bank that won't change anything is incompatible with the GDPR."

    Reply

  20. José Ramírez says:

    JCS wrote:
    No one—no one—is going to be confused when they see “Jose Ramirez” instead of “José Ramírez”
    but the lawyers among us still think it’s a critical issue. This is the very apotheosis of a first world
    problem.

    Maybe you dont care, but I do!

    Reply

  21. says:

    @blog Update: it turns out that EBCDIC supports code pages INCLUDING one with all the diacritical marks the bank claimed it was impossible to support! It's been available since an update to the standard in the mid-1980s! (EBCDIC code page 435.)

    There's also UTF-EBCDIC, which allows EBCDIC to encode all valid Unicode character code points—over a million of them.

    Verdict: the bank in question has an incompetent IT department.

    | Reply to original comment on wandering.shop

Trackbacks and Pingbacks

  1. :

    EU 内の銀行で、氏名のアルファベットにグレーブ(á)とかウムラウト(ü)とか、アクセント記号がついている人が本名を登録できない、という問題に対する訴訟があり、アクセント記号が登録できないのはGDPR(一般データ保護規則)違反である、という判決が2019年に出ていたそうです。
    テレンス・エデン氏(Terence Eden)のブログによれば、銀行がアクセント記号つきの氏名登録を拒絶したのに対し、この顧客はEU一般データ保護規則(GDPR)の16条「訂正の権利」を根拠に訴えるに至ったそうです。
    この16条は以下のようなもの。

    The data subject shall have the right to obtain from the controller without undue delay the rectification of inaccurate personal data concerning him or her. Taking into account the purposes of the processing, the data subject shall have the right to have incomplete personal data completed, including by means of providing a supplementary statement.
    データ主体は、管理者から、不当に遅滞することなく、自己と関係する不正確な個人データの訂正を得る権利を有する。取扱いの目的を考慮に入れた上で、データ主体は、補足の陳述を提供する方法による場合を含め、不完全な個人データを完全なものとさせる権利を有する。
    一般データ保護規則(仮訳)

    これに対し銀行は、1995年に開発された米国製メインフレーム上のアプリを使っており、このシステムが1964-65 に制定されたEBCDICコード表を使っていることから技術的に対応不可能なのだ、と弁明していたということ。

    実際のところ、EBCDIC にもCode page 37などアクセント記号に対応したセットがあり、これを使うように設計しなかったシステム設計の問題のようですが、EU内の人の移動が今ほど活発でなかった時代、アクセント記号を使わない国の銀行では少数の外国名顧客のことまで考えていなかったのでしょうね。
    本来の正しい文字で氏名を登録できない、という意味では漢字かなを使ってる日本人なんかは絶対にヨーロッパの銀行では登録できないだろうと思いますけど、それはまあ日本がEU加盟国ではないのでそこまでは要求されることはなさそうです。でもEU域内の加盟国で使われる文字に非対応なのはだめでしょうね。
    今ならUnicodeベースで作るでしょうから、EU向けに提供するサービスでアクセント記号を氏名に入れられないなんてことはなかなか起こらないとは思います。しかし、うっかり入力文字のフィルタでA-Za-zなんてことをしていると、こういう苦情が来る可能性は十分にありそうです。
    判決が話題になったのは今ですが、判決自体は2019年に出ていたものです。銀行は25年前のシステムを直せたんでしょうかね?
    via Twitter
    共有:TwitterFacebook

  2. :

    Article 16 of the GDPR says, among other things, that

    The data subject shall have the right to obtain from the controller without undue delay the rectification of inaccurate personal data concerning him or her.

    A normal person would take that to mean you have the right to have substantive errors in your records corrected. Things like age, arrest record, employment record and so on. Errors that can cause actual harm if not corrected. A Brussels court has ruled that it also means that European Union citizens also have the right to have their name represented correctly.
    This is why everyone hates lawyers. The issue is that many names have diacritical marks—Erdős or Ceaușescu for example—and older computer systems are unable to represent them. This is a particular issue in the banking industry that, sadly, is often still running their businesses on last-century IBM mainframes that use the EBCDIC character encoding system.
    It’s hard to imagine today but bytes weren’t always 8 bits, and ASCII, let alone UTF-8, wasn’t always the way characters were encoded. EBCDIC was IBM’s system and as the 800 pound gorilla they saw to it that EBCDIC’s use was widespread. The problem is that its standard code page had no diacritical marks so José Ramírez is going to have his name encoded as “Jose Ramirez”.
    Someone with way too much time on his hands decided that was intolerable and sued under Article 16. Now the Court of Appeals of Brussels has agreed and demanded that the bank fix things. It’s hard to imagine a quick fix for this and it’s even harder to see how the fix is not going to cost hundreds of thousands of dollars. Of course lawyers don’t know or care about any of this. They just want it fixed. Now. That’s why we all hate them.
    If I were the bank, I’d suggest that this particular customer find another bank but the lawyers probably won’t like that solution either. Yes, yes. I know I’m intolerant. I know I’m not a European. I know my name doesn’t have a diacritical—but it’s misspelled so often it might as well have. But, you know, a little common sense would be appropriate here. No one—no one—is going to be confused when they see “Jose Ramirez” instead of “José Ramírez” but the lawyers among us still think it’s a critical issue. This is the very apotheosis of a first world problem.

What are your reckons?

All comments are moderated and may not be published immediately. Your email address will not be published.Allowed HTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <p> <pre> <br> <img src="" alt="" title="" srcset="">