Adding Sign Language to HTML5 video

I watched this video from my colleagues in NHS England - it's the first time I've seen a Sign Language overlay on a Twitter video.

NHS

@NHSuk

Need help fast, but not sure what to do? Go straight to 111.nhs.uk .
To find out more about NHS 111 including how to use the NHS 111 BSL interpreter service visit nhs.uk/111 #NHS111 pic.x.com/tk18uvm2vy

---

Is it possible to have multiple Sign Languages available to a video?

What you may not know is that British Sign Language and American Sign Language are completely different languages. Not like written English and American English with the occasional difference in spelling, but mutually incomprehensible.

Wouldn't it be nice if users could flip between different Sign Languages as easily as different subtitles?

HTML5 gives us the power to add multiple subtitles to a video. For example:

 HTML<video>

   <source src="example.mp4" type="video/mp4">

   <track label="English" kind="subtitles" srclang="en" src="en.vtt" default>

   <track label="Español" kind="subtitles" srclang="es" src="es.vtt">

</video>

How could we do the same for Sign Language?

Overlays

The aim is to overlay one video on top of another. That is, have the main video in the background and the the person signing in front.

Here's a simple way to do it in HTML. Two videos, in a <div>.

 HTML<div class="video-container">

   <video class="main" autoplay>

      <source src="video.mp4" type="video/mp4">

   </video>

   <video class="sign" autoplay>

      <source src="bsl.mp4"   type="video/mp4">

   </video>

</div>

Then some CSS to layer one on top of the other:

 CSS.video-container {

  position:relative;

  width:400px;

}

.main {

    height: 200px;

    width: 400px;

    float: left;

    position: absolute; 

}

.sign {

    width: 100px;

    position: absolute;

    bottom:-200px;

    right:0px;

}

That gives you this effect:

There's deliberately no audio, and the Signing is not related to the main video. This is just an experiment. My CSS may break in unusual ways!

Source videos Creative Commons - How to greet someone in English and Learn Basic Greetings in British Sign Language.

Three points to note:

1. There's no synchronisation - both are set to autoplay, but some JS magic will be needed to keep them both in sync, or to pause at the same time.
2. No way to switch between different Sign Languages. Again, will need some JS to do that.
3. No transparency.

The last one is very annoying.

Alpha Video

Most Signers are usually "green screened" - that is, the background image is transparent and they are seamlessly overlayed onto the video.

HTML5 videos don't have an "alpha" channel. That is, no transparency. Chrome had an experimental alpha flag - but that doesn't seem to have gone very far.

There are some hacky open source plugins, and a few experiments using canvas - but nothing mainstream.

Separate Video Tracks

The final alternative is to produce multiple videos, each with a different Sign Language superimposed. In which case, you'd need a small user interface asking the user which video they want to see.

Standards

I do some work on web standards - but I'm ignorant about this area. It looks like there was some attempts at standardising having multiple video tracks, or improving the accessibility of web video - but they don't seem to have got much further.

I suppose what I'm looking for is:

 HTML<video controls>

  <source src="video.mp4" type="video/mp4">

  <video label="British Sign Language" kind="overlay" srclang="sgn-GB" src="bsl.mp4">

  <track label="English" kind="subtitles"  srclang="en" src="en.vtt" default>

  <audio label="English" kind="Commentary" srclang="en" src="en.mp3" default>

</video>

Would that be useful, I wonder?