Figment API

How to work with Figment.

General Functions

figment.activate(configs? : object)

Activates Figment and returns an output MediaStream or MediaStreamTrack. Accepts an optional configs object to set Figment's initial state. Possible configurations are described under General Configs below.

Note: Depending on whether an input MediaStream or video MediaStreamTrack is specified, the function respectively returns an output MediaStream or a MediaStreamTrack.

Example:

let inputVideoTrack = await getVideoTrack()

let outputVideoTrack = await figment.activate(
    {
      activeApp: "VB",
      background: {
        type: "blur",
        blurStrength: "balanced",
      },
      inputFrequency: 1000 / 25,
      vbQuality: "performance",
      maskBehavior: 'tight',
      showFPS: true,
      inputMediaStream: {
        videoTrack: inputVideoTrack,
        stopTracksOnDestruction: false
      }
    }
  );

returns: Promise<MediaStream | MediaStreamTrack>

figment.deactivate()

Deactivates Figment. All rendering and computations are stopped.

If stopTracksOnDestruction in the inputMediaStream config is set to true, Figment stops the input MediaStream or video MediaStreamTrack when deactivated.

returns: Promise

figment.setInputMediaStream(mediaStream: MediaStream, stopTracksOnDestruction: boolean = false)

Sets the MediaStream that Figment uses as video input and returns the corresponding output MediaStream. The parameter stopTracksOnDestruction specifies whether the input MediaStream will be stopped when deactivating Figment.

Usage:

inputStream = await this.getMediaStream()
let outputStream = await figment.setInputMediaStream(inputStream);

This function is meant to be executed while Figment is running (e.g. when one of your users changes the camera). For setting the initial MediaStream please use the inputMediaStream field of the configs parameter in the activate() function.

returns: Promise<MediaStream>

figment.setInputVideoTrack(videoTrack: MediaStreamTrack, stopTracksOnDestruction: boolean = false)

Sets the MediaStreamTrack that Figment uses as video input and returns the corresponding output MediaStreamTrack. The parameter stopTracksOnDestruction specifies whether the input MediaStreamTrack will be stopped when deactivating Figment.

Usage:

inputVideoTrack = await this.getVideoTrack()
let outputVideoTrack = await figment.setInputVideoTrack(inputVideoTrack);

This function is meant to be executed while Figment is running (e.g. when one of your users changes the camera). For setting the initial videoTrack please use the inputMediaStream field of the configs parameter in the activate() function.

returns: Promise<MediaStreamTrack>

figment.setOption(option: string, value: string | boolean)

General function to change the internal state of Figment (see General Configs). This function is meant to be used while Figment is running. All of these can also be set during activation with the configs parameter of the activate() function.

option: Option to be set. Available options are: set_active_app | show_hide_performance | set_input_frequency

• set_active_app — Sets the activeApp config field and returns a new output MediaStream. Note: Depending on whether an input MediaStream or video MediaStreamTrack is specified (i.e. in the activate ,setInputMediaStream or setInputVideoTrack function), this function either returns an output MediaStream or a MediaStreamTrack. value: VB (default) | AR. returns: MediaStream | MediaStreamTrack.

• show_hide_performance — Shows and hides FPS display by setting the showFPS config field. value: 'show'/true | 'hide'/false (default)

• set_input_frequency — Sets the inputFrequency config field. value: 1 - Number.MAX_SAFE_INTEGER, default: 1000/30

value: value specifying the set option.

returns: Promise

General Configs

Configs to change Figment's internal state. The configs given below can be set during activation with the activate() function or while Figment is running with the setOption() function.

activeApp : 'VB' (default) | 'AR' — Specifies which app should be used. Virtual background (VB) or augmented reality (AR).

background : object — Specifies the virtual background. Following configs are available:

background: {
  type: 'image' | 'blur',
  bgImageUrl?: './path/to/image.jpg',
  blurStrength?: 'weak' | 'balanced' (default) | 'strong'
},

arFilter : string | object — Specifies the augmented reality filter. The value can either be a string specifying the arFilterId, or an object with the fields arFilterId and animateFilters . animateFilters is an object that allows specifying animationType and objectCount for our animated filters.

We offer the following filters:

On face

classic_sunglasses | retro_sunglasses | golden_sunglasses | mustache | devil_horns | clown_nose | crown | floral_crown | birthday_hat | pirate_hat | joker_mask

Above Face

mario_question_box | sims_icon

Around Face (animated)

{
    arFilterId: "exclamation_mark" | "light_bulb" | "question_mark"
    animateFilters: {
        animationType: "rotate_360" | "rotate_180" | "floating" | ""
        objectCount : number // only with animationType "floating"
    }
}

inputFrequency : number — Lets you steer the frequency of frames entering the machine learning model. To give you maximum control, this function lets you specify the exact interval between frames in milliseconds. The higher the interval, the lower Figment's CPU load and output FPS. To specify the input frames per second (FPS) you can set the value to 1000/FPS. The inputFrequency parameter is a great tool to balance FPS with CPU load (see Performance and CPU Load).

Note: Only the input FPS is specified, but not Figment's final output FPS. This is because the output FPS always depends on the inference time of the machine learning model, which varies depending on the performance of the local machine.values: 1 - Number.MAX_SAFE_INTEGER, default: 1000/30

inputMediaStream : object — Lets you set the MediaStream that Figment uses as video input. You can either specify a full MediaStream or an individual video MediaStreamTrack. The parameter stopTracksOnDestruction specifies whether the input MediaStream will be stopped when deactivating Figment. Note: When a complete MediaStream is specified, Figment removes its audio track to avoid unwanted auditory side effects. We therefore recommend that you provide Figment with a video MediaStreamTrack or a copy of your desired input MediaStream.

inputMediaStream: {
  mediaStream?: MediaStream,
  videoTrack?: MediaStreamTrack
  stopTracksOnDestruction: boolean
}

vbQuality : string — Specifies the machine learning model used for the virtual background feature. The performance setting emphasizes computational efficiency. Slight degradation in segmentation quality is expected. The quality setting emphasizes segmentation fidelity. A slight drop in FPS is expected. value: performance | quality (default)

maskBehavior : string — Allows to specify a tight or a slightly looser mask. Both masks aim to accurately segment the person. The loose mask shows small fractions of the background in order to preserve the whole person, while the tight mask aims to show as little background as possible with the possibility that the mask may cut into the person at times. value: tight (default) | loose

showFPS : string | boolean — Allows to show and hide a performance <div> that we append to document.body which displays the frames per second (FPS) achieved by the respective feature (i.e. 'VB' or 'AR'). Note that the final FPS always depends on the specified inputFrequency. In most cases, higher FPS can be achieved by increasing the input frequency at the expense of a higher CPU load. value: 'show'/true | 'hide'/false (default)

If no configs are specified the defaults are used.

Virtual Background

figment.setOption(option: string, value: string)

Setting Figment's internal state for the virtual background feature (see General Configs). This function is meant to be used while Figment is running.

option: Option to be set. Available options are: set_background_img | blur_background | activate_quality_mode | activate_performance_mode | set_mask_behavior

• set_background_img — Changes the current background image. Sets background.type='image' and background.bgImageUrl=value. value: ./path/to/image.jpg

• blur_background — Blurs the background. Sets background.type='blur' and background.blurStrength=value. We offer three different blur strengths: weak, balanced and strong. value: weak | balanced (default) | strong

• set_mask_quality — Sets the field vbQuality config field and returns a new output MediaStream. Depending on whether an input MediaStream or video MediaStreamTrack is specified (i.e. in the activate ,setInputMediaStream or setInputVideoTrack function), this function either returns an output MediaStream or a MediaStreamTrack. value: performance | quality (default). returns: MediaStream | MediaStream.

• set_mask_behavior — Sets the maskBehavior config field. value: tight (default) | loose

value: value specifying the set option.

returns: Promise

Usage examples:

let outputStream = await figment.setOption('set_mask_quality', 'performance')

await figment.setOption('set_background_img', './path/to/image.jpg')

Augmented Reality

figment.setOption(option: string, value: string | object)

Setting Figment's internal state for the augmented reality feature (see General Configs). This function is meant to be used while Figment is running.

option: Option to be set. Available options are: set_ar_filter

• set_ar_filter — Lets you set the config field arFilter. values: (see General Configs).

value: value specifying the set option.

returns: Promise

Usage examples:

await figment.setOption('set_ar_filter', "classic_sunglasses");

await figment.setOption(
    'set_ar_filter', 
    {
        arFilterId: 'question_mark',
        animateFilters: {animationType: 'floating', objectCount: 3}
});