Adding Sign Language to HTML5 video

by @edent | # # # | 5 comments | Read ~262 times.

I watched this video from my colleagues in NHS England - it's the first time I've seen a Sign Language overlay on a Twitter video.

Is it possible to have multiple Sign Languages available to a video?

What you may not know is that British Sign Language and American Sign Language are completely different languages. Not like written English and American English with the occasional difference in spelling, but mutually incomprehensible.

Wouldn't it be nice if users could flip between different Sign Languages as easily as different subtitles?

HTML5 gives us the power to add multiple subtitles to a video. For example:

<video>
   <source src="example.mp4" type="video/mp4">
   <track label="English" kind="subtitles" srclang="en" src="en.vtt" default>
   <track label="Español" kind="subtitles" srclang="es" src="es.vtt">
</video>

How could we do the same for Sign Language?

Overlays

The aim is to overlay one video on top of another. That is, have the main video in the background and the the person signing in front.

Here's a simple way to do it in HTML. Two videos, in a <div>.

<div class="video-container">
   <video class="main" autoplay>
      <source src="video.mp4" type="video/mp4">
   </video>
   <video class="sign" autoplay>
      <source src="bsl.mp4"   type="video/mp4">
   </video>
</div>

Then some CSS to layer one on top of the other:

.video-container {
  position:relative;
  width:400px;
}
.main {
    height: 200px;
    width: 400px;
    float: left;
    position: absolute; 
}
.sign {
    width: 100px;
    position: absolute;
    bottom:-200px;
    right:0px;
}

That gives you this effect:


There's deliberately no audio, and the Signing is not related to the main video. This is just an experiment. My CSS may break in unusual ways!

Source videos Creative Commons - How to greet someone in English and Learn Basic Greetings in British Sign Language.

Three points to note:

  1. There's no synchronisation - both are set to autoplay, but some JS magic will be needed to keep them both in sync, or to pause at the same time.
  2. No way to switch between different Sign Languages. Again, will need some JS to do that.
  3. No transparency.

The last one is very annoying.

Alpha Video

Most Signers are usually "green screened" - that is, the background image is transparent and they are seamlessly overlayed onto the video.

HTML5 videos don't have an "alpha" channel. That is, no transparency. Chrome had an experimental alpha flag - but that doesn't seem to have gone very far.

There are some hacky open source plugins, and a few experiments using canvas - but nothing mainstream.

Separate Video Tracks

The final alternative is to produce multiple videos, each with a different Sign Language superimposed. In which case, you'd need a small user interface asking the user which video they want to see.

Standards

I do some work on web standards - but I'm ignorant about this area. It looks like there was some attempts at standardising having multiple video tracks, or improving the accessibility of web video - but they don't seem to have got much further.

I suppose what I'm looking for is:

<video controls>
  <source src="video.mp4" type="video/mp4">
  <video label="British Sign Language" kind="overlay" srclang="sgn-GB" src="bsl.mp4">
  <track label="English" kind="subtitles"  srclang="en" src="en.vtt" default>
  <audio label="English" kind="Commentary" srclang="en" src="en.mp3" default>
</video>

Would that be useful, I wonder?

5 thoughts on “Adding Sign Language to HTML5 video

  1. Sumantro Das says:

    This concept looks like a good fit for machine-learning models like Adobe to transcode audio into visual signs.

    1. @edent says:

      No, that's a terrible idea. Pay highly skilled professionals.

      Take a look at these and tell me how long before a machine can replicate what these people do:


      https://www.youtube.com/watch?v=jIyU8RZu1gc

      1. Ben Smith says:

        Also, sign language needs to be translated – it is not just a visual representation of the spoken word. That’s why it’s offered in addition to subtitles.

  2. Ben Smith says:

    I am curious of what the advantages of the overlay approach are beyond scaling the person doing the signing according to the viewer's preference?

    If you are going to the effort / expense of producing a signed version of a video, why not output both the original and as many signed versions as you need with the signer 'baked in' and allow the viewer to switch between videos for the version they prefer. That would also work across most devices already.

    1. @edent says:

      Yup. That's probably the sensible answer.
      But I do like the idea of scaling the Signer - or adding multiple sign languages after the fact. If you have X spoken languages and Y sign languages, you could end up with a large number of permutations.
      As I said, just a little experiment to see if it would work.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.