VLC and YouTube will only play "Equirectangular" videos in spherical mode. So, how to convert a dual fisheye to equirectangualr?

The Simple Way

ffmpeg \
  -i original.mp4 \
  -vf "v360=input=dfisheye:output=equirect:ih_fov=189:iv_fov=189" \
  360.mp4

However, this has some "quirks".

The first part of the video filter is v360=input=dfisheye:output=equirect - that just says to use the 360 filter on an input which is dual fisheye and then output in equirectangular.

The next part is :ih_fov=189:iv_fov=189 which says that the input video has a horizontal and vertical field of view of 189°. That's a weird number, right?

You'd kind of expect each lens to be 180°, right? Here's what happens if :ih_fov=180:iv_fov=180 is used:

The lenses overlaps a little bit. So using 180° means that certain portions are duplicated.

I think the lenses technically offer 200°, but the physical casing prevents all of that from being viewed. I got to the value of 189° by trial and error. Mostly error! Using :ih_fov=189:iv_fov=189 get this image which has less overlap:

It isn't perfect - but it preserves most of the image coherence.

Cut Off Images

There's another thing worth noticing - the top, right, bottom, and left "corners" of the circle are cut off. If the image sensor captured everything, the resultant fisheye would look something like this:

I tried repaging the video to include the gaps, but it didn't make any noticeable difference.

Making Equirectangular Videos Work With VLC

Sadly, ffmpeg will not write the metadata necessary to let playback devices know the video is spherical. Instead, according to Bino3D, you have to use exiftool like so:

exiftool \
        -XMP-GSpherical:Spherical="true" \
        -XMP-GSpherical:Stitched="true" \
        -XMP-GSpherical:ProjectionType="equirectangular" \
        video.mp4

Putting It All Together

The LG 360 records audio in 5.1 surround using AAC. That's already fairly well compressed, so there's no point squashing it down to Opus.

The default video codec is h264, but the picture is going to be reprojected, so quality is always going to take a bit of a hit. Pick whichever code you like to give the best balance of quality, file size, and encoding time.

Run:

ffmpeg \
  -i original.mp4 \
  -vf "v360=input=dfisheye:output=equirect:ih_fov=189:iv_fov=189" \
  -c:v libx265 -preset fast -crf 28 -c:a copy \
  out.mp4; exiftool \
        -XMP-GSpherical:Spherical="true" \
        -XMP-GSpherical:Stitched="true" \
        -XMP-GSpherical:ProjectionType="equirectangular" \
        out.mp4

That will produce a reasonable equirectangular file suitable for viewing in VLC or in VR.

If this has been useful to you, please stick a comment in the box!

This upset some fans.

The thing about widescreen is that it chops off the top and bottom of the picture. The top of Hawkeye's head? Gone! The bottom of Hot-Lips' boots? Vanished! אױ װײ איז מיר

But, there is an upside. Or should that be side-side? Anyway, the new remaster adds some previous unseen detail to the left and right of the picture.

Here's a grab from the original:

And here's the remastered wide version:

Perhaps an animated GIF will help?

As you can see, the remaster has great picture quality and bonus content at the sides. However, you might also be able to notice that the widescreen image has been stretched a fair bit. By my calculation it was originally a 1.72 aspect ratio before been distorted to 1.777.

This GIF makes it a bit more obvious what has been stretched and how the image has moved from the centre:

So, is it possible to take the sides of the widescreen version and stitch them on to the edges of the 4:3 version? Yes!

With a bit of ffmpeg magic, I scaled the 16:9 version back to its original ratio, then I added some black padding around it, then I overlaid the 4:3 on top - giving it some transparency so you can see how well it fits.

The (frankly overwhelming) command is:

ffmpeg -i wide.mp4 -i orig.mp4 \
   -filter_complex \
   "[0]scale=670:390,setsar=1,pad=670:480:(ow-iw)/2:(oh-ih)/2:black,setpts=PTS-0.375/TB[v0];[1]scale=640:480,setsar=1,format=rgba,colorchannelmixer=aa=0.5[fg];[v0][fg]overlay=(W-w)/2:(H-h)/2[vid];[0:a][1:a]amerge=inputs=2[aout]" \
   -map "[vid]" -map "[aout]" -c:v copy -c:a aac -strict experimental -b:a 192k copy output.mp4

That also shifts the first video by a couple of frames due to the files being slightly out of synch. I've also kept both soundtracks, for your discordant joy.

Here it is - the intro to M*A*S*H with extra side bits:

As you can see, they adjusted the font size of the main titles. As the video progresses, you'll notice the vertical offset changes - I don't know how much manual adjustment was made. The videos shift in and out of synchronisation - I assume the original was transferred from film to tape in a less precise process than the remaster.

ToDo

• Get AI to fill in the missing corners?
• Get AI to continually correct the placement of the video?
• Get AI to add a laugh track?

Add the file to an MKV

Use MKVmerge:

mkvmerge "audio.wav" --chapters "audio.cue" -o "audio.mkv"

You can see that chapter names have been added to the .mkv if you run ffmpeg -i or mkvinfo.

Split the MKV by chapter

This generates one file per chapter:

mkvmerge -D -S "audio.mkv" --split chapters:all -o "split-%02d.mkv"

The -D switch means no video will be copied. -S means no subtitles.

Extract the audio

This quickly pulls out the audio, raw, without changing anything.

mkvextract split-01.mkv tracks "0:output"

The raw audio is placed in a file called output

You can also do this with:

ffmpeg -i "split-01.mkv" -acodec copy "whatever.wav"

Replace .wav with .ac3 or .flac or whatever the audio format is. You can transcode the audio to something else. I recommend Opus

ffmpeg -i "split-01.mkv" -af aformat=channel_layouts="7.1|5.1|stereo" -b:a 1536k "whatever.opus"

There is a bug with the surround sound mapping in ffmpeg.

Rename it

This is bit convoluted. ffprobe can get the chapter name:

ffprobe -i "split-01.mkv" -show_chapters -loglevel error

The use of -loglevel error means you only get the data, rather than the debug cruft.

In order to get the title, we get ffprobe to spit out JSON and then get jq to interpret it:

ffprobe -i "split-01.mkv" -print_format json -show_chapters -loglevel error | jq ".chapters[0].tags.title" -r

All in one script to extract the audio and rename it

#!/bin/bash
AUDIO="audio.wav"
CUE="audio.cue"
MKV="audio.mkv"

# What's the audio format?
FORMAT=$(ffprobe -v error -select_streams a:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 "$AUDIO")

# Create the MKV
mkvmerge "$AUDIO" --chapters "$CUE" -o "$MKV"

# How many chapters are there?
JSON=$(ffprobe -i "$MKV" -print_format json -show_chapters -loglevel error)
COUNT=$(echo $JSON | jq ".chapters | length" )

# Split the MKV into chapters
mkvmerge -D -S "$MKV" --split chapters:all -o "split-%02d.mkv"

# Loop through all the created .mkv files
COUNTER=1
while [ $COUNTER -le $COUNT ]; do

  # Zero pad the counter
  printf -v ZEROTRACK "%02d" $COUNTER

  # Get the chapter name
  JSON=$(ffprobe -i "split-$ZEROTRACK.mkv" -print_format json -show_chapters -loglevel error)
  TITLE=$(echo $JSON | jq ".chapters[0].tags.title" -r)
  FILENAME="[$ZEROTRACK] $TITLE.$FORMAT"

  # Raw Audio
  mkvextract "split-$ZEROTRACK.mkv" tracks "0:$FILENAME"
  # Transcode. Optional
  #ffmpeg -i "split-$ZEROTRACK.mkv" "$FILENAME.opus"

  let COUNTER=COUNTER+1 
done

# Delete the generated MKVs. Optional
#rm *.mkv

Note, there is no single command in MKVtools to do this