Using YouTube to Transcode Videos to DASH on the Command Line
This is part of my redecentralisation efforts to liberate my videos from YouTube.
MPEG-DASH is a simple method of streaming videos which doesn't require any specialised server software. You convert a high resolution video into a series of smaller resolution videos. You chop each of the videos up into several chunks. As the video plays, your browser then decides which chunk of the video to load next depend on the bandwidth available to it. Easy!
Aim
From the command line:
- Upload a video to YouTube
- Download transcoded video in several formats suitable for MPEG DASH Streaming
- Generate DASH manifest
- Generate the HTML5 markup
Uploading
There are a couple of YouTube video uploaders. These are useful if you've got a video on a server, or don't want to waste a browser window.
- porjo/youtubeuploader has binaries available for Linux/Mac/Windows.
- tokland/shoogle a Python interface for all of Google's APIs.
I'm using Porjo's client. After following the authentication instructions, the output is something like:
./youtubeuploader_linux_amd64 -filename testing.mp4
Uploading file 'testing.mp4'...
Progress: 1.37 Mbps, 9852099 / 9852099 (100.000%) ETA 0s
Upload successful! Video ID: 6NV5heuaVuc
Downloading
The only program worth using is youtube-dl.
We can see every video format that YouTube has created using:
youtube-dl -F 6NV5heuaVuc
That will return something like:
format code | extension | resolution | note |
---|---|---|---|
139 | m4a | audio only DASH audio | 49k , m4a_dash container, mp4a.40.5@ 48k (22050Hz) |
140 | m4a | audio only DASH audio | 128k , m4a_dash container, mp4a.40.2@128k (44100Hz) |
134 | mp4 | 640x360 | DASH video 104k , avc1.4d401e, 30fps, video only |
160 | mp4 | 256x144 | DASH video 108k , avc1.4d400b, 30fps, video only |
133 | mp4 | 426x240 | DASH video 242k , avc1.4d400c, 30fps, video only |
135 | mp4 | 854x480 | DASH video 1155k , avc1.4d4014, 30fps, video only |
137 | mp4 | 1920x1080 | DASH video 2306k , avc1.640028, 30fps, video only |
136 | mp4 | 1280x720 | DASH video 2310k , avc1.4d4016, 30fps, video only |
17 | 3gp | 176x144 | small , mp4v.20.3, mp4a.40.2@ 24k |
36 | 3gp | 320x180 | small , mp4v.20.3, mp4a.40.2 |
43 | webm | 640x360 | medium , vp8.0, vorbis@128k |
18 | mp4 | 640x360 | medium , avc1.42001E, mp4a.40.2@ 96k |
22 | mp4 | 1280x720 | hd720 , avc1.64001F, mp4a.40.2@192k (best) |
It may take several minutes for YouTube to generate all the video formats - the length of time will depend on the length and quality of the video.
We want to download all the DASH videos and the DASH audios. We also want the subtitles - even if they are autogenerated. Finally, we want the thumbnail image for the HTML5 poster.
The youtube-dl
command to download all the DASH audio is:
youtube-dl -f 'all[vcodec=none]' -o '%(title)s-%(format)s.%(ext)s' 6NV5heuaVuc
To get all the DASH videos:
youtube-dl -f 'all[acodec=none]' -o '%(title)s-%(format)s.%(ext)s' 6NV5heuaVuc
This will create a separate MP4 video for each resolution, and separate audio file for each bitrate.
To generate a single thumbnail, run:
youtube-dl --write-thumbnail --skip-download 6NV5heuaVuc
To download the subtitles:
youtube-dl --write-auto-sub --sub-format vtt --write-sub 6NV5heuaVuc
Convert to DASH
For this step, we need the open source Bento4 toolkit.
The video should already have "fragments" - the chunks which make up the stream - you can check this with:
./mp4info "Video-133 - 426x240 (DASH video).mp4"
In the output, you should see:
Movie:
duration: 0 ms
time scale: 90000
fragments: yes
The audio downloaded from YouTube will not have fragments, so you'll need to run this command on every audio file you've downloaded:
./mp4fragment "Video-140 - audio only (DASH audio).m4a" high.m4a
Combine and create manifest
Let's take those audio and video files and make ourselves a manifest!
./mp4dash -o output-video 133.mp4 135.mp4 136.mp4 137.mp4 160.mp4 high.m4a low.m4a
This creates the folder output-video
and creates folders for the chunked audio and video.
It will also produce the MPD - a manifest file which references all of the chunks.
We're nearly ready to play back.
Play DASH Video in the browser
As far as I can tell, no browser natively supports MPEG-DASH. You will need to use the latest version of dash.js to make everything work smoothly.
Add this piece of JavaScript to your page:
<script src="https://cdn.dashjs.org/latest/dash.all.min.js"></script>
Then reference the MPD in a video
element:
<video width="1080" data-dashjs-player src="stream.mpd" controls poster="example.jpg">
<track kind="subtitles" src="en.vtt" srclang="en" label="English">
</video>
Demo
Here is an adaptive stream of one of my videos - hosted natively on this site, with no server-side streaming.
Problems?
Fallback video
Some browsers can't play back DASH, even with the JavaScript - so you can put a fallback video in the HTML.
<video width="1080" data-dashjs-player controls >
<source src="stream.mpd" type="application/dash+xml">
<source src="fallback.mp4" type="video/mp4">
</video>
HLS
Because Apple are special, they don't support DASH on its phones. So you'll need to use their proprietary HLS streaming solution. You can use Bento4 to create HLS in exactly the same way as you create DASH.
You may also need to use hls.js JavaScript to get things to work smoothly.
Subtitles
You can use DASH native subtitles, or HTML5 subtitles.
I found that the HTML5 subtitles downloaded by youtube-dl
contain some strange CSS which didn't work well with the DASH video. You may need to test to see how it works for you.
All in one command?
Hahahaha! Not yet! If you know of a good way to do this - let me know!
Raphael says:
Great...but I don't understand why you want to use Youtube in the first place? You can convert any avi directly to MPREG-DASH yourself by using https://github.com/Cloudoki/mp4-to-mpegdash This is the all in one command you wanted 😉 Cheers!
Terence Eden says:
Because I have a weak and feeble laptop which takes ages to convert videos. Using other people's computers is cheaper 🙂
rahuldottech says:
Whoa, this is amazing! I really like how the browser decides which subsequent chunk of the video to load next depending on the bandwidth available. The video played super smoothly on my super crappy internet connection (I live in a developing country), which is impressive.
It's also interesting how in chrome "right-click > save video as..." is disabled, but I guess that's to be expecting considering how this works.
Sucks how Safari on iOS doesn't support it though, although HLS seems easy enough to implement.
wristy says:
Hi, thanks for article, very interesting. I have installed youtube-dl and upload, then I uploaded a mp4 file which is about 30MB in size, the very famous big_buck_bunny.mp4. It's been more than 30 minutes since i have uploaded the file, and when i executed youtube-dl -F "filename.mp4" it says:
[youtube] Yc74rOEtCFY: Downloading webpage [youtube] Yc74rOEtCFY: Downloading video info webpage ERROR: Yc74rOEtCFY: YouTube said: This video is unavailable.
@edent says:
That video doesn't exist -
If you're having problems with youtube-dl you can get support at https://github.com/rg3/youtube-dl/issues/
wristy says:
That's interesting ! I clicked on the link you provided and i'm seeing my video. But when i execute the command line
C:\Program Files\YoutubeDL>youtube-dl -F Yc74rOEtCFY [youtube] Yc74rOEtCFY: Downloading webpage [youtube] Yc74rOEtCFY: Downloading video info webpage ERROR: Yc74rOEtCFY: YouTube said: This video is unavailable.
...The vid is not available. Thanks for you help, I'll try contact YouTube.
Regards
@edent says:
Have you set the video to private? Or not published it?
wristy says:
Hi, sorry i'm not uploading every day videos so I did not even notice is on Private. I change to Public but I think it will take a while to process. You should be able to view it shortly.
wristy says:
Quick update, I am able to see all the information about the video now, using the command line C:\Program Files\YoutubeDL>youtube-dl -F Yc74rOEtCFY
It just took a whole lot of time to process the video. Thanks again.
Regards,