performs

https://github.com/upf-gti/performs

Last synced: 6 months ago · JSON representation ·

Repository

Basic Info

Host: GitHub
Owner: upf-gti
License: apache-2.0
Language: JavaScript
Default Branch: main
Homepage: https://performs.gti.upf.edu/
Size: 334 MB

Statistics

Stars: 11
Watchers: 4
Forks: 0
Open Issues: 1
Releases: 0

Created over 2 years ago · Last pushed 8 months ago

Metadata Files

Readme License Citation

Performs

Performs is designed to visualize and synthesize humanoid animations for customized avatars. It supports two types of animations: keyframe animations (glTF, BVH) and script animations (SiGML, BML). The second mode integrates a variety of BML (Behavior Markup Language) instructions for Manual Features (MF) and Non-Manual Features (NMF) to create cohesive animations, performed in real-time. The project has made significant progress in synthesizing NMF, which is essential for enhancing the realism of sign language animations. This initiative began as part of the SignON project, a user-centric and community-driven effort aimed at facilitating communication among Deaf, hard of hearing, and hearing individuals across Europe.

Alt Text

The application allows users to customize the following features: - Avatar: - [x] Character - [x] Cloth color - Background: - [x] Space: Open space, Studio, Photocall - [x] Color - [x] Photocall image - Illumination: - [x] Light position - [x] Light color

It can be easily integrated into other applications by using inserting an HTML iframe or by downloading the code.

Performs as an Iframe

Insert Performs inside your application using the iframe HTML element. html <iframe src='https://performs.gti.upf.edu'>

[!IMPORTANT]
You can customize the default settings combining the available options with multiple parameters by concatenating them with &.

So for example showing a custom avatar in a blue chroma would look like that: html <iframe src='https://performs.gti.upf.edu?avatar=https://models.readyplayer.me/67162be7608ab3c0a85ceb2d.glb&background=studio&color=rgb(54,54,190)'>

Installation and Running

Clone the repository: git clone https://github.com/upf-gti/performs.git To run locally, host a server from the main project folder. You can also use the build version without GUI.

[!IMPORTANT]
To load the default avatar, an internet connection is required. If you prefer to work offline, you can change the modelToLoad in the init() method of Performs.js. You can use the resources from the /data folder or add your own.

You can load the customization features by appending the URL params outlined in the Integration Guide or by loading a JSON file (which can be exported from the Performs app) with containing the same parameters. Then, call the setConfiguration(options) function from Performs passing the JSON object with the options as a parameter.

Project structure

Each project must include at least one HTML file for webpage definition and the JavaScript build files to execute Performs. - index.html ```html <!DOCTYPE html> My app using Performs

    <script src="../external/jquery-3.6.0.min.js"></script>
    <script async src="../external/es-module-shims.js"></script>

    <script type="importmap">
        {
            "imports": {
                "three": "../external/three/build/three.module.js",
                "three/addons/": "../external/three/jsm/"
            }
        }
    </script>
</head>
<body>
    <div id="loading" class="modal" style="background-color:rgba(51, 51, 51, 0.5); position: absolute; width: 100%; height: 100%; 
            display: flex; justify-content: center; align-items: center; z-index: 100; color: white; font-size: larger;">
        <p>Loading avatar, please wait...</p>
    </div>

    <script type="module" src="main.js"></script>

</body>

``-main.js`

By default, Performs generates the code for creating a fully functional web application with its built-in GUI (GUI.js, style.css, and lexgui.js library are required). However, the application can also be used without the GUI or with a custom interface. Configuration options can be set through the GUI, by loading a JSON file, or by appending URL parameters. For a complete list of available options, please refer to the Integration Guide.

```javascript import { Performs } from 'Performs.js'

const performs = new Performs();

// Load customization options file fetch('performsSettings.json') .then(response => response) .then(response => response.json()) .then(options => { // Init performs with configuration options performs.init(options);
}) .catch(error => console.error(error)); ```

The following sections provide examples of how to initialize the application without a GUI or configure it to initialize with a specific mode as the default.

Keyframe animation

This mode visualizes clip animations across different avatars. To ensure effective retargeting, certain options need to be adjusted for the source animation and the target avatar: - Embedded transforms: Retargeting only takes into account the transforms from the actual bone objects (local transforms). If set to true, external (parent) transforms are computed and embedded into the root joint (only once, on construction). Afterwards, parent transforms/matrices can be safely modified and will not affect in retargeting. Useful when it is easier to modify the container of the skeleton rather than the actual skeleton in order to align source and target poses. - true or false

Reference pose mode: Determines which pose will be used as the retargeting bind pose.
- DEFAULT: Uses skeleton's actual bind pose
- CURRENT: Uses skeleton's current pose
- TPOSE: Forces the bind pose to be a T-pose and sets CURRENT mode.

A detailed explanation of these options can be found in the retargeting repository.

[!IMPORTANT]
Currently only .gltf, .glb and .bvh are supported. If you happen to use another format, please convert it to the mentioned formats.

Code example

main.js

By default, Keyframe mode is enabled. However, if a configuration file for Script mode is detected, the mode will automatically switch. When multiple animations are loaded, a crossfade blending is applied during transitions between animations. The blend time can be adjusted using the blendTime attribute of keyframeApp.

```javascript import { Performs } from 'Performs.js'

const performs = new Performs();

// Set customization options const options = {
srcEmbeddedTransforms: true, trgEmbeddedTransforms: true, srcReferencePose: 0, // DEFAULT trgReferencePose: 2, // TPOSE autoplay: true, animations = [{name: './data/animations/myAnimation.glb', data: null}], // Set keyframe animation files' URL. 'data' can be an already parsed file

onReady = () => { // Function called after loading the application
    // Change to Keyframe mode 
    performs.changeMode(Performs.Modes.KEYFRAME);

    // Play the animation after 1s
    setTimeout(() => performs.keyframeApp.changePlayState(true), 1000);
}

}

// Init performs with the options performs.init(options); ```

Script animation

This mode allows for the generation of procedural animations based on instructions. The system interprets the instructions based in SiGML (Signing Gesture Markup Language), and translates them into an extended version of BML (Behavior Markup Language), or directly into BML in JSON format. While the results may not match the quality of keyframe animations, this approach requires neither expensive equipment nor extensive manual labor, making it highly suitable for scalable sign synthesis. These instructions can be written manually or generated and exported using Animics' script mode, which enables the creation and editing of this type of animation through a visual timeline-clip representation.

The current supported instructions are explained in detail in BML Instructions. An example: javascript { type: "faceLexeme", start: 0.1, attackPeak: 0.6, relax: 1.5, end: 1.8, amount: 0.1, lexeme: "ARCH" }

Code example

main.js

Include ./data/dictionaries/ folder in your project to use the Idle animation and the NGT gloss dictionary.

``` javascript import { Performs } from 'Performs.js'

const performs = new Performs();

const options = { config = './data/configFile.json', // Set required config file for Script mode

scripts = [ // Set script instruction files' URL and/or parsed data
    {
        type: 'sigml', 
        name: './data/animations/scriptAnimation.sigml'
        },
    {
        type: 'bml', 
        data: {
            behaviours: [{
                type: "faceLexeme",
                start: 4,
                attackPeak: 0.6,
                relax: 1.5,
                end: 1.8,
                amount: 1,
                lexeme: "ARCH"
            }]
        }
    },
],
applyIdle: true, // Enable Idle animation to blend with recieved instructions
onReady = () => { // Function called after loading the application
    setTimeout(() => {
        // Play the animation after 1s
        performs.scriptApp.replay();
    }, 1000)                    
}

};

// Init performs with the options performs.init(options); ```

Examples

Some examples on simple NGT (Dutch Sign Language)

Kind lezen boek (child reads a book): Alt Text

Man rijden fiets (Man rides a bicycle) : Alt Text

Architecture

The realizer is divided into main sections, each containing several files. The whole pipeline is warpped inside the CharacterController class which is in charge of receiving a BML block and triggering and executing its instructions when required.

Controllers section

Files in this block: CharacterController, FacialController and BodyController

CharacterController: is the main entry. Everything is controlled from here. Every instance of FacialController, BodyController, BehaviourManager and BehaviourPlanner is automatically created and managed by this class. Users only need to instantiated this class and call its functions:
- start
- update
- processMsg: receives all the instructions of a block and triggers the whole pipeline to be synthesise the actions specified.
FacialController: manages the blending of the diferent animations involved in the face including facial gestures, eye movement and head displacement.
BodyController: manages the blending of the diferent animations involved in the body including trunk, shoulders, arms, hands and fingers movement.

BML section

The files in this block: BehaviourManager, BehavhourPlanner and BehaviourRealizer

BevahiourPlanner: automatically generates some instructions such as blinking
BehaviourManager: deals with all instruction blocks. Is in charge of triggering instructions when their time comes.
BehaviourRealizer: a set of diverse functionalities to execute some particular instruction

Adding avatars

[!IMPORTANT]
Currently only .gltf and .glb are supported. If you happen to use another format, please convert it to the mentioned formats.

To add a new avatar to Performs, you must follow this steps:

Make sure the avatar is rigged (if it is not rigged, we recommend using Mixamo)
Check that your avatar has the correct scale and orientation.
Use the performs-atelier tool to configure all the parameters needed for the application (only required for Script mode), this will generate a configuration .json file containing all the needed information.

Collaborators

Jaume Pozo @japopra
Víctor Ubieto @victorubieto
Eva Valls @evallsg
Carolina del Corral @carolinadcf
Pablo García @PZerua
Alex Rodríguez @jxarco

Acknowledgments

Three.js - An open-source JavaScript library for creating interactive 3D and 2D graphics in web browsers.
Lexgui.js - A simple and lightweight GUI library for creating graphical user interfaces for web applications.

We would like to extend our gratitude to the creators and maintainers of these libraries for their invaluable contributions to the open-source community.

Support

This project is being developed with partial financial support of:

| EMERALD Project (2023-2026) | SignON Project (2021-2023) | | --- | --- | | miciu | logomaxr |

Owner

Name: UPF-GTI
Login: upf-gti
Kind: organization
Email: gti-info@llista.upf.edu
Location: Universitat Pompeu Fabra, Barcelona

Website: https://www.upf.edu/web/gti
Repositories: 10
Profile: https://github.com/upf-gti

Interactive Technologies Group

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Pozo"
  given-names: "Jaume"
- family-names: "Ubieto"
  given-names: "Víctor"
- family-names: "Valls-Garolera"
  given-names: "Eva"
- family-names: "del Corral"
  given-names: "Carolina"
- family-names: "Rodríguez"
  given-names: "Alejandro"
title: "Performs"
version: 1.0.0
date-released: 2023
url: "https://github.com/upf-gti/performs"

GitHub Events

Total

Watch event: 8
Delete event: 1
Push event: 83
Create event: 1

Last Year

Watch event: 8
Delete event: 1
Push event: 83
Create event: 1

Committers

Last synced: 11 months ago

All Time

Total Commits: 366
Total Committers: 9
Avg Commits per committer: 40.667
Development Distribution Score (DDS): 0.571

Past Year

Commits: 127
Committers: 5
Avg Commits per committer: 25.4
Development Distribution Score (DDS): 0.189

Top Committers

Name	Email	Commits
japopra	6****a	157
evalls	e**4@g**m	122
Carolina del Corral	7****f	28
Eva Valls	e**s@w**u	28
victorubieto	v**1@g**m	17
japopra	j**o@w**u	7
PinyaColada	n**i@g**m	4
u149957	u**7@u**u	2
PZerua	p**f@g**m	1

Committer Domains (Top 20 + Academic)

webglstudio.gti.upf.edu: 2 upf.edu: 1

Issues and Pull Requests

Last synced: 11 months ago

All Time

Total issues: 1
Total pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Total issue authors: 1
Total pull request authors: 0
Average comments per issue: 0.0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 1
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 1
Pull request authors: 0
Average comments per issue: 0.0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

performs

Science Score: 62.0%

Repository

Basic Info

Statistics

Metadata Files

README.md

Performs

Performs as an Iframe

Installation and Running

Project structure

Keyframe animation

Code example

Script animation

Code example

Examples

Architecture

Controllers section

BML section

Adding avatars

Collaborators

Acknowledgments

Support

Owner

Citation (CITATION.cff)

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Dependencies