Virtual Display Use Case Exploration

A Collection of Interesting Ideas

Second Screen WG/CG virtual meeting

8 June 2022

Anssi Kostiainen

Virtual Display?

"a display instance without a physical connection to the graphics card"

"a software representation of a physical display"

UX Considerations

• Display Arrangement
  • OS
• Display Source Picker
  • getDisplayMedia()
  • HTMLMediaElement.captureStream()?
• Virtual Display Creation
  • OS
• Window Placement
  • OS
  • Multi-Screen Window Placement API?
    • (Virtual workspaces/desktops a non-goal, how about Virtual Displays?)

Display Arrangement

Display Source Picker

(Chrome display source picker UI, https://www.webrtc-experiment.com/getDisplayMedia/)

Virtual Display Creation

(Deskreen Virtual Display Picker UI, https://deskreen.com/)

getDisplayMedia UI vs Deskreen Virtual Display UI

Window Placement

Implicit consent example: drag and drop

OSes have a variety of implicit consenting mechanisms

• Keyboard shortcuts in Windows: "Windows logo key + Shift + Left arrow or Right arrow to move an app or window in the desktop from one monitor to another."

• Drag and drop evolved in macOS: "Drag the app window you want to move to the edge of the screen [or] enter Mission Control, then drag the window up to the space you want to use."

Could such implicit permission grants work for web app window placement?

The traditional Web API approach: user's consent via browser prompt

OS-level: controls built into window manager

Browser prompt vs OS controls

Goal: Enable UX innovation and protect the user's privacy

How? Set a strong privacy baseline & leave room for Quality of Implementation differentiation?

Virtual Display support for the web

Ideation

Goals

• "Let me create a new virtual display surface & place windows on it, get its relative positioning and a representation I can pass on to other Web APIs."
• MediaStream as the "common currency" for getDisplayMedia & WebRTC interop
• Reuse existing API constructs and browser and OS UI/UX

Constraints

• Agnostic with regard to the display technology
• Meet security and privacy expectations

Assumptions

• Web users expect multi-screen use cases to work similarly with Virtual Displays
• Web APIs shouldn't(?) fully abstract away virtual/physical display boundary

Requirements (to make this a 1st class web capability)

• A new virtual display surface type that maps to an OS-level concept
• Allow {extending|mirroring} of {monitor|window|browser} display surface to a virtual display surface
• User-initiated and programmatic window placement
• Use platform-native UX concepts (e.g. drag/move windows across, share controls)

Related work

• Platform-level: open-source projects (Deskreen), mainstream product features (Sidecar) and more...

Related, but different use cases

• Present a new document (Presentation API) or video element (Remote Playback API)
• Share controls across devices: Synergy/Barrier, Logitech Flow, Universal Control
• Admin/advanced: various Remote desktop software, PowerToys FancyZones layouts

Deskreen Experiments

Use Case : Browser of the tablet as an ad-hoc Virtual Display

"Sometimes I want to use the display of my tablet along with my primary device's display and let web apps make use of both. I want to be able to control what web content to present on my tablet."

Download video as .mp4

Use Case : TV as an ad-hoc Virtual Display

"Sometimes when lounging in the living room I want to extend my workspace to the TV. I want to stay in control of what web content to share on the TV to a larger audience."

Download video as .mp4

Prototype Implementation Details

1. Deskreen mirrors a virtual display (created with BetterDummy) to a browser window on a tablet

• WebRTC MediaStream
• Same LAN

Deskreen Architecture

See also: session initiation sequence diagram

Prototype Implementation Details

1. Deskreen mirrors a virtual display (created with BetterDummy) to an offscreen browser window
2. Presentation API mirrors that offscreen browser window to a TV connected to a Chromecast (1-UA mode triggered via browser UI)

• WebRTC MediaStream Cast Mirroring
• Chromecast in same LAN

Design Noodling

Display Arrangement

OS & Platform APIs - built-in feature, some API gaps

• Windows "multiple desktops" / VirtualDesktopManager
• macOS "spaces" / no public API
• Chrome OS "virtual desks" / Wayland API issues

Web APIs

• Virtual screen arrangement readable via getScreenDetails()

Display Source Picker

Web APIs

• getDisplayMedia() satisfies "display surface to MediaStream" req / screen mirroring/sharing use case
• Virtual display surface type
• Picker UI for virtual displays (create a new?)

getDisplayMedia extension ideas (reusable for picking multiple virtual displays?):

• getDisplayMedia issue #204: Concurrent capture of multiple surfaces
• Capture multiple surfaces w/ getDisplayMedia proposal

Virtual Display Creation

OS & Platform APIs - "advanced users only"

• Windows: user-mode indirect display driver model
• macOS: 3rd party tools e.g. BetterDummy
• Linux: e.g. GNOME headless native backend in Mutter
• Generic: physical Dummy Display Plugs, various Virtual Display drivers

Web APIs

• Virtual Display concept does not exists
  • Interesting parallels: Audio Output Devices API to allow routing audio to a "virtual" (other than the default) audio sink, see also new sink available

API Ideation

Reusable concepts

• display surface and its DisplayCaptureSurfaceType
• CursorCaptureConstraint
• virtual screen arrangement, isExtended, getScreenDetails()

getDisplayMedia extensions

• Picker UI extended to allow a new Virtual Display creation
• Extend DisplayCaptureSurfaceType enum with virtual
• Reuse existing signaling of closed/minimized display surfaces
• ...

IDL sketch

Considered Alternatives

New "getDisplaySurface" API

• Redundant if getDisplayMedia extensions satisfy reqs

Presentation API extension

• Presentation API Presentation of objects proposal incl. MediaStream
• Site Initiated Mirroring API
  • getDisplayMedia extension: "no API that allows a MediaStream to be sent directly to a secondary display (without going through WebRTC)."

Remote Playback API extensions

• Remote Playback API via HTMLMediaElement.srcObject = ms too much indirection

Possible path from an Exploration onwards

•   Explainer

  • Use cases, reqs, API sketch, considered alternatives     We are here!

•   Explainer review

  • Web community, browser & OS vendors, hardware ecosystem

•   Draft web spec

  • W3C Second Screen CG in collaboration with WebRTC WG?

•   Prototype in code

  • Example: Chromium launch process

•   Prototype feedback

  • Web developer feedback can change the shape of the API significantly

•   Standards track

  • Subject to support, possible WG adoption

</html>