# Video360 Sample

Video360 is a hybrid Daydream app that includes a 2D (non-VR)
Activity and a Daydream (VR) Activity capable of viewing 360 media.

- The 2D (non-VR) Activity uses standard Android \& OpenGL APIs to render 360
  content and can be used as a standalone 360 media viewer.

- The Daydream (VR) Activity uses the same rendering system as the 2D (non-VR)
  Activity but also uses the Daydream SDK to render the scene in VR. It renders
  a floating 2D (non-VR) UI in VR with a custom Android View and allows the
  user to control a running video with the Daydream Controller.

The Video360 sample app is included in the Google VR SDK for Android. If you
have not yet downloaded the SDK, see [Getting Started](https://developers.google.com/vr/develop/android/get-started)
to set up and start navigating the SDK in Android Studio.

Video360 is more complex than the [`VrPanoramaView`](https://developers.google.com/vr/android/reference/com/google/vr/sdk/widgets/pano/VrPanoramaView)
and
[`VrVideoView`](https://developers.google.com/vr/android/reference/com/google/vr/sdk/widgets/video/VrVideoView)
[VR View](https://developers.google.com/vr/develop/android/vrview) based sample apps also included in the SDK. These samples are lightweight
widgets that can be added to existing Android apps to display 360 images or
video. They have a simple API and limited room for customization.

## Device video support and performance

All [Daydream-ready devices](https://vr.google.com/daydream/smartphonevr/phones/)
support rendering video at 3840 x 2160 at 60 fps. Video playback capabilities and
decoding support on other devices varies and should be tested to ensure that
video renders correctly and that performance is acceptable.

- Daydream-ready Android devices can render [H264-encoded](https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC)
  video at 3840 × 2160 and 60 fps. For top-bottom stereo, the video frame should
  contain a pair of 2880 x 1440 scenes stacked on top of each other resulting
  in a 2880 x 2880 video stream.

- Lower-end Android devices cannot render at this resolution, but most
  devices can render 1920 x 1080 at 60 fps.

## Overview of functionality

### Embedding a 360 viewer in Android Activities

Video360 demonstrates how to create a 360 media viewer
using standard Android \& OpenGL APIs. The sample has an embedded
[GLSurfaceView](https://developer.android.com/reference/android/opengl/GLSurfaceView.html)
that renders a 3D scene. This functionality doesn't use any Daydream-specific
APIs and can be used to create a pure 2D (non-VR) app to view 360 media.
![](https://developers.google.com/static/vr/android/samples/video360.jpg)   
*The 2D (non-VR) mode for the sample, showing the default panorama used if no media is specified. See [UI and adb commands to load media](https://developers.google.com/vr/android/samples/video360#ui_and_adb_commands_to_load_media) for more information. *

The core of the app involves rendering a video on a 3D sphere.

1. The embedded `MonoscopicView` class uses [Android's OpenGL
   APIs](https://developer.android.com/guide/topics/graphics/opengl.html) to render a 3D mesh. This is a standard `GlSurfaceView` implementation for OpenGL.
2. The mesh has an OpenGL [GL_TEXTURE_EXTERNAL_OES](https://www.khronos.org/registry/OpenGL/extensions/OES/OES_EGL_image_external.txt) which allows the system to render video textures.
3. An Android [SurfaceTexture](https://developer.android.com/reference/android/graphics/SurfaceTexture.html) is created using this OpenGL texture.
4. This `SurfaceTexture` is wrapped in an [Surface](https://developer.android.com/reference/android/view/Surface.html) which is a low-level Android object for quickly moving a stream of images through the system.
5. This `Surface` is sent to a [MediaPlayer](https://developer.android.com/reference/android/media/MediaPlayer.html) which renders video to it.

The sample also demonstrates how to render 360 images or draw arbitrary images
using a
[Canvas](https://developer.android.com/reference/android/graphics/Canvas.html)
onto this `Surface`.

See [Android's Graphics
Architecture](https://source.android.com/devices/graphics/) for low-level
details about how Surfaces and their related components work.

This Activity uses Android's [Sensor
API](https://developer.android.com/guide/topics/sensors/sensors_motion.html) to
reorient the scene based on the orientation of the phone. It also uses touch
input to allow the user to manipulate the camera.

### Creating a 360 video viewer for Daydream

Video360 demonstrates how to render 360 media
inside a Daydream Activity. The same rendering system for the embedded view is
used together with `GvrActivity`. However, instead of using
`MonoscopicView`'s sensor and rendering systems, the Daydream API's sensor and
rendering system is used.
![](https://developers.google.com/static/vr/android/samples/video360vr.jpg)   
*The Daydream mode for the sample. The **congo.mp4** video from the git repository is being played.*

### Rendering a 2D (non-VR) UI inside a VR scene

The Daydream Controller is used to interact with a 2D (non-VR) UI rendered inside the VR
scene.

1. Similar to the sphere used for the video above, an OpenGL mesh is used for the floating UI. This mesh has an an OpenGL [`GL_TEXTURE_EXTERNAL_OES`](https://www.khronos.org/registry/OpenGL/extensions/OES/OES_EGL_image_external.txt) which is wrapped in a `SurfaceTexture` and then wrapped in a `Surface`.
2. [Surface.lockCanvas()](https://developer.android.com/reference/android/view/Surface.html#lockCanvas(android.graphics.Rect)) is used to acquire an Android `Canvas`.
3. This `Canvas` is passed to the [ViewGroup.dispatchDraw()](https://developer.android.com/reference/android/view/ViewGroup.html#dispatchDraw(android.graphics.Canvas)) of the `VideoUiView` class.
4. The `VideoUiView` uses Android standard 2D (non-VR) rendering system to draw on the given `Canvas`.
5. Controller click events are translated into 2D (non-VR) events and dispatched to the underlying `View`. `CanvasQuad.translateClick` performs a simple hitbox calculate and generates an XY coordinate that the Android touch system uses to generate standard click events.

### Permissions and hybrid VR app flow

The Video360 sample can load images or videos from the device's external
storage. This requires asking the user to grant the
[READ_EXTERNAL_STORAGE](https://developer.android.com/reference/android/Manifest.permission.html#READ_EXTERNAL_STORAGE)
permission. Since the user can launch the VR Activity prior to granting this
permission. The sample demonstrates how to use the
[DaydreamApi.exitFromVr()](https://developers.google.com/vr/reference/android/com/google/vr/ndk/base/DaydreamApi#exitFromVr(android.app.Activity,%20int,%20android.content.Intent))
API to switch from the VR Activity to the 2D (non-VR) Activity, [request a runtime
permission](https://developer.android.com/training/permissions/requesting.html),
and then return to VR using
[DaydreamApi.launchInVr()](https://developers.google.com/vr/reference/android/com/google/vr/ndk/base/DaydreamApi#launchInVr(android.content.Intent)).

The sample also allows the user to toggle between 2D (non-VR) and VR modes using the UI.

## UI and adb commands to load media

The Video360 sample is meant to be used as a basis for other apps rather than as
a standalone video player so it has a minimal UI. After the app is installed and
the user accepts permissions, it will render a placeholder panorama and allow
the user to switch in and out of VR. When in VR, the user can use the Daydream
Controller to interact with a floating 2D (non-VR) UI.

Loading real media requires using **adb** to send the appropriate `Intent` to
the app. Two example commands are given below. They assume you've used **adb
push** to push **IMAGE.JPG** and **VIDEO.MP4** to the device at **/sdcard/**.

- Load a top-bottom stereo image in the VR Activity:

      adb shell am start \
      -a android.intent.action.VIEW \
      -n com.google.vr.sdk.samples.video360/.VrVideoActivity \
      --ei stereoFormat 2 \
      -d "file:///sdcard/IMAGE.JPG"

- Load a monoscopic video in the 2D (non-VR) Activity:

      adb shell am start \
      -a android.intent.action.VIEW \
      -n com.google.vr.sdk.samples.video360/.VideoActivity \
      --ei stereoFormat 0 \
      -d "file:///sdcard/VIDEO.MP4"

See [`MediaLoader`'s Javadoc](https://github.com/googlevr/gvr-android-sdk/blob/master/samples/sdk-video360/src/main/java/com/google/vr/sdk/samples/video360/MediaLoader.java#L42) for more information about how to craft the
appropriate intent.

The sample uses Android's built-in media decoding system so it is capable of
rendering any [supported media
format](https://developer.android.com/guide/topics/media/media-formats.html).

Many devices support more formats. For images, most Android devices can render
4096 x 4096 bitmaps.

## Core components

- `VideoActivity` - a standard Android Activity. This hosts the 2D (non-VR) UI for the 360 viewer. This also demonstrates how to request permissions and launch a Daydream Activity.
- `MonoscopicView` - a `GLSurfaceView` that renders the 3D scene inside `VideoActivity`. It uses the [Sensor
  API](https://developer.android.com/guide/topics/sensors/sensors_motion.html) to adjust the OpenGL camera's orientation based on the phone's orientation. It also has a `TouchTracker` component that allows the user to change the camera's pitch \& yaw by dragging on the screen.
- `SceneRenderer` and `Mesh` - the core OpenGL rendering components. `Mesh` generates and renders an equirectancular sphere that displays the video. `SceneRender` coordinates the entire rendering process in 2D (non-VR) and VR.
- `MediaLoader` - this parses the `Intent` and loads the file. It uses `MediaPlayer` to render video and `Canvas` to render images.
- `VrVideoActivity` - a [GvrActivity](https://developers.google.com/vr/android/reference/com/google/vr/sdk/base/GvrActivity) that handles the Daydream-specific functionality. It uses the Daydream Controller API to let the user interact with the 2D (non-VR) UI while they are in VR.
- `CanvasQuad` - a minimal system to render 2D (non-VR) UI inside OpenGL scenes. In combination with `VideoUiView`, this class can render arbitrary Android `View`s to an OpenGL texture and then render that texture in the 3D scene. It also has basic functionality to convert Daydream Controller events into 2D (non-VR) Android touch events.

## Extending the Video360 sample

Video360 includes basic functionality for rendering 360 media. There are several
ways to extend the sample:

- **Improve the sensor system in `MonoscopicView`.** The sample demonstrates
  basic sensor input. It doesn't have any sensor fusion beyond what Android
  provides. Depending on the use case, it may be useful to incorporate [sensor
  fusion](http://smus.com/sensor-fusion-prediction-webvr/) to combine multiple
  sensors and inputs when manipulating the camera.

- **Extend the functionality of `TouchTracker`.** `TouchTracker` only allows
  the user to manipulate the pitch \& yaw of the camera. The design also uses
  [Euler angles](https://en.wikipedia.org/wiki/Euler_angles) to manipulate the
  camera. This can lead to strange interactions when the touch input is mixed
  with the gyroscope input. A better system would be to use
  [quaternions](https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation)
  to processing the rotation. The touch events can then be used to drag the
  camera using [spherical linear
  interpolation](https://en.wikipedia.org/wiki/Slerp) which allows for more
  complex motion. However, the exact UI depends on the needs of the app that
  will embed this input system. It is also possible to use [multi-touch
  gestures](https://developer.android.com/training/gestures/scale.html) to
  allow the user to zoom in. This involves detecting gestures and modifying
  the field of view of the OpenGL camera.

- **Improve media support.** `MediaLoader` only has support for basic video
  files using `MediaPlayer`. A real application would use a more complex media
  system such as [ExoPlayer](https://google.github.io/ExoPlayer/guide.html) to
  render video. ExoPlayer can render to a `Surface` just like `MediaPlayer`
  but it has support for [many more
  formats](https://google.github.io/ExoPlayer/supported-formats.html)
  including streaming video. You can also use the [Grafika](https://github.com/google/grafika)
  samples if you want to build your own media decoding pipeline.

- **Add more 2D (non-VR) UI in VR.** `VideoUiView` shows a minimal UI for playing
  video. However, the general design behind it can be extended to create more
  complex UIs. `CanvasQuad` demonstrates how to render an arbitrary Android
  `View` inside an OpenGL scene and allow the user to interact with the `View`
  via a Daydream Controller. Almost any Android `View` can then be embedded
  inside this root `View` which allows for complex 2D (non-VR) UIs inside VR. A complex
  app can use a
  [WebView](https://developer.android.com/guide/webapps/webview.html) to
  embedd arbitrary HTML \& Javascript inside a VR app. This makes it easy to
  create complex 2D (non-VR) UIs inside VR.

- **Support more 360 media formats.** Only
  [equirectangular](https://en.wikipedia.org/wiki/Equirectangular_projection)
  media is supported in the sample. This is a very common format for 360
  content, but it is inefficient. It wastes pixels at the poles of the scene
  and has low resolution at the equator. The spherical mesh used to rended
  this content is also inefficient due to the high number of verticies used
  near the poles. A better format is the [equal area
  projection](https://www.blog.google/products/google-vr/bringing-pixels-front-and-center-vr-video/)
  which improves the visual quality of the media for a given file size.
  However, most media processing system do not yet have support for this
  format.