This page introduces how to integrate and use the FaceUnity AR Filter extension in your Android or iOS app.
The FaceUnity AR Filter extension encapsulates the core APIs of the FaceUnity Nama SDK. By calling the setExtensionProperty or setExtensionPropertyWithVendor method of the Agora Video SDK v4.0.0 and passing in the corresponding key
and value
, you can quickly integrate FaceUnity's AR filter capabilities.
Taking setExtensionProperty
as an example, the key
is named after the FaceUnity API, and the value
wraps either certain or all of the parameters of that API in JSON. When you call setExtensionProperty
and pass in the pair of key
and value
, it is equivalent to calling the corresponding FaceUnity API. The case is the same for setExtensionPropertyWithVendor
.
The FaceUnity AR Filter extension works together with the Agora Video SDK v4.x. Refer to the following docs to integrate the SDK and start a basic video call:
You need to activate the extension on Agora Console. Save the certificate file you obtain from activation. The certificate is used in subsequent integration.
Android
.aar
files to your project folder (for example, the /app/libs
path).authpack.java
to where the app module is located. (For example, the package name in the sample code is io.agora.rte.extension.faceunity.example
, so the certificate file should be saved to /app/src/main/java/io/agora/rte/extension/faceunity/example
)./app/src/main/assets
path). For details, see Resource package structure.app/build.gradle
file, add the following line in dependencies
:implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])
import io.agora.rtc2.Constants;
import io.agora.rtc2.IMediaExtensionObserver;
import io.agora.rtc2.IRtcEngineEventHandler;
import io.agora.rtc2.RtcEngine;
import io.agora.rtc2.RtcEngineConfig;
import io.agora.rtc2.video.VideoCanvas;
import io.agora.rte.extension.faceunity.ExtensionManager;
iOS
// A sample project folder
.
├── <ProjectName>
├── <ProjectName>.xcodeproj
└── Resource
.framework
files to the project folder. Ensure that you select Embed & Sign instead of Embed.authpack.h
to the project folder (for example, the <ProjectName>
path).Resource
path). For details, see Resource package structure. #import <AgoraRtcKit/AgoraRtcEngineKit.h>
#import "authpack.h"
This section describes the call sequence of using the extension. For a detailed parameter description, see the API Reference.
Android
After creating and initializing RtcEngine
, call enableExtension
before other APIs (including enableVideo
and joinChannel
):
private void enableExtension(boolean enabled) {
// Initialize ExtensionManager before calling enableExtension
ExtensionManager.getInstance(mRtcEngine).initialize(this);
mRtcEngine.enableExtension("FaceUnity", "Effect", enabled);
}
iOS
After creating and initializing AgoraRtcEngineKit
, call enableExtensionWithVendor
before other APIs (including enableVideo
and joinChannelByToken
):
[self.agoraKit enableExtensionWithVendor:@"FaceUnity"
extension:@"Effect"
enabled:self.enable];
After receiving the onExtensionStarted
callback, call setExtensionProperty
(Android) or setExtensionPropertyWithVendor
(iOS), and pass in the corresponding keys and values:
fuSetup
and the value as the pointer to the certificate file.fuLoadAIModelFromPackage
and the value as the path of the AI capability model file ai_xxx.bundle
and the AI capability type.Take Android as an example:
private void initExtension() {
// Initialization
try {
JSONObject jsonObject = new JSONObject();
JSONArray jsonArray = new JSONArray();
for (byte it : authpack.A()) {
jsonArray.put(it);
}
jsonObject.put("authdata", jsonArray);
setExtensionProperty("fuSetup", jsonObject.toString());
} catch (JSONException e) {
Log.e(TAG, e.toString());
}
// Loads the AI model
File modelDir = new File(getExternalFilesDir("assets"),
"face_unity/model/ai_face_processor.bundle");
try {
JSONObject jsonObject = new JSONObject();
jsonObject.put("data", modelDir.getAbsolutePath());
jsonObject.put("type", 1 << 10);
setExtensionProperty("fuLoadAIModelFromPackage", jsonObject.toString());
} catch (JSONException e) {
Log.e(TAG, e.toString());
}
}
// Only updates the key and value when calling setExtensionProperty
private void setExtensionProperty(String key, String property) {
mRtcEngine.setExtensionProperty("FaceUnity", "Effect", key, property);
}
Call setExtensionProperty
(Android) or setExtensionPropertyWithVendor
(iOS), and pass in the corresponding keys and values. You can implement the following functions:
You can call the method as needed. For a full list of keys and values, see the API Reference.
When you do not need the extension, follow these steps to release the resources:
setExtensionProperty
(Android) or setExtensionPropertyWithVendor
(iOS) and pass in the key as fuDestroyLibData
.fuDestroyLibData
callback, call destroy
to destroy RtcEngine
(Android) or AgoraRtcEngineKit
(iOS).The complete sample code and project is provided on GitHub:
Platform | Language | Sample project |
---|---|---|
Android | Java | FaceUnity/android |
iOS | Objective-C | FaceUnity/ios |
Android
Clone the repository:
git clone https://github.com/AgoraIO-Community/AgoraMarketPlace.git
On Agora Console, download the Android package of FaceUnity AR Filter.
Unzip the package, and save all .aar
files to the FaceUnity/android/app/libs
path.
Contact Agora to get the certificate file and resource package.
Save the certificate file authpack.java
to FaceUnity/android/app/src/main/java/io/Agora/rte/extension/faceunity/example
.
Save the required model and prop files from the resource package to FaceUnity/android/app/src/main/assets/face_unity
under the project folder.
Open the sample project FaceUnity/android
in Android Studio.
Sync the project with the Gradle files.
Open the FaceUnity/android/app/src/main/java/io/Agora/rte/extension/FaceUnity/example/Config.java
file, and replace <YOUR_APP_ID>
with your App ID. To get an App ID, see Get Started with Agora.
public interface Config {
String mAppId = "<YOUR_APP_ID>";
String mToken = null;
}
Connect an Android device (not an emulator), and run the project.
iOS
Clone the repository:
git clone https://github.com/AgoraIO-Community/AgoraMarketPlace.git
On Agora Console, download the iOS package of FaceUnity AR Filter.
Contact Agora to get the certificate file and resource package.
Save the certificate file authpack.h
to FaceUnity/ios/ExtensionExample
.
Save the required model and prop files from the resource package to FaceUnity/ios/Resource
under the project folder.
In Terminal, go to the FaceUnity/ios directory
, and run the following command to install dependencies using CocoaPods:
pod install
Open the project FaceUnity/ios/ExtensionExample.xcworkspace
in Xcode.
Unzip the downloaded iOS package, and import all .framework
files. Ensure that you select Embed & Sign instead of Embed.
Open the Config.h
file, and replace <YOUR_APP_ID>
with your App ID. To get an App ID, see Getting Started with Agora.
NSString *const appID = @"<YOUR_APP_ID>";
NSString *const token = nil;
Connect an iOS device (not an emulator), and run the project.
If successful, the sample project is installed on your Android or iOS device.
disableExtension
, enableAITracking
, setComposer
, and setSticker
buttons on the user interface.enableAlTracking
to enable
face, gesture, and body recognition. You can see the recognition results in the upper right corner.setComposer
to set the beauty effect, and drag the Color Level or Filter Level to adjust the beautification intensity.setSticker
to load the props. The default is a cat ear effect.For a detailed parameter description, see the API Reference.
For a full description of the FaceUnity resource package, refer to:
1. Why there is no beauty effect when I run the sample project?
Common reasons include:
2. Why it takes nearly 1 second to render the video after the camera is enabled?
Loading FaceUnity resources can cause openGL rendering to freeze and costs around 800ms. Agora recommends that you use a workaround in your app (for example, add a loading screen).
3. Why enableExtension/enableExtensionWithVendor returns -3?
-3
is returned when the extension package cannot be found. Ensure that you save the package according to Integrate the extension.
4. Why enableExtension/enableExtensionWithVendor returns 0, but then setExtensionProperty/setExtensionPropertyWithVendor returns -2 and there is no beauty effect?
This is because enableExtension/enableExtensionWithVendor
is called improperly. Ensure that you follow the call sequence in Enable the extension.