MS-UFAD: A Large-Scale Dataset for Real-world Unified Face Attack Detection with Text Descriptions

FSGAN:FSGAN performs quite well in face swapping, effectively preserving the identity features of the source face while transferring facial and expression information to the target image. It introduces a novel face reconstruction method based on Recurrent Neural Networks (RNNs), which can adjust for changes in pose and expression and can be applied to single images or video sequences. For video sequences, a continuous interpolation method based on ghosting, Delaunay triangulation, and barycentric coordinates is introduced. FSGAN contains many different sub-networks suitable for more general face swapping tasks. However, the authors did not specify how to train such a large network. Nevertheless, one can take inspiration from the ideas presented in the article or apply parts of the network to achieve good results. In 2022, FSGANv2 was proposed, offering a facial swapping solution that does not require subject-specific training, achieving significant changes in pose and expression through iterative deep learning, suitable for single images or video sequences. It uses a facial blending network to maintain target skin color and lighting conditions while dealing with occluded areas. However, for comparative effectiveness, we implemented the method using the official FSGAN code available at: https://github.com/NVlabs/imaginaire.

PBE:Most face swapping methods utilize GANs; however, this paper employs a diffusion model for editing. We use it for face swapping, where PBE achieves this by utilizing self-supervised training to disentangle and reorganize source images and templates. Simple methods, though, can lead to noticeable blending artifacts. The authors thoroughly analyze this issue and propose content bottlenecks and strong augmentation to avoid simple copy-and-paste solutions of template images. To ensure the controllability of the editing process, the authors designed arbitrary-shaped masks for template images and used classifier-free guidance to increase similarity with the template images. The entire framework requires only a single forward pass of the diffusion model without any iterative optimization. It performs well on in-the-wild data and achieves high-fidelity controlled editing. However, after swapping, there may occasionally be other artifacts. We employ the official implementation code: https://github.com/Fantasy-Studio/Paint-by-Example.

SimSwap:SimSwap is a current state-of-the-art (SOTA) face swapping method, introducing a new approach to address the lack of generalization to arbitrary identities. It transfers identity information at the feature level through an ID injection module while using a weak feature matching loss to preserve the attributes of the target face. This method overcomes challenges in generalization capabilities and attribute preservation found in traditional face swapping methods, enabling arbitrary source-to-target face swapping while maintaining attributes like expressions and gaze directions. SimSwap offers single-frame and video face swapping modes and supports specific person swapping. Many face swapping tools use it as their internal swapping algorithm. However, it failed to preserve facial expressions and gaze directions, possibly because the ID extraction was not clean. We use the official implementation code: https://github.com/neuralchen/SimSwap.

InSightFace:InSightFace is a powerful face algorithm that has been continuously updated and iterated in recent years, serving as a tool used within many models. Some use it as their internal basic face swapping method, while others leverage its face detection functionality. Now available as a package, InSightFace can be used conveniently and quickly. It seamlessly blends the features of the source face while maintaining the target facial identity characteristics. We use the official implementation code: https://github.com/deepinsight/insightface.

SAFA: Building upon FOMM, SAFA improves to tackle challenges caused by complex scene structures in facial animation tasks. This method constructs specific geometric structures to simulate different components of facial images, aiming to solve potential improper distortions and occlusions in generated images. At the core of the method is the use of a 3D Morphable Model (3DMM) to simulate the face, multiple affine transformations to simulate other foreground components like hair and beard, and an identity transformation for the background. The geometric embedding of 3DMM not only helps generate realistic structures for the driving scene but also better perceives occluded areas in the generated images. Incorporating 3DMM has led to a qualitative leap in maintaining facial rigidity during expression-driven animations, meaning less facial deformation and simultaneous driving and swapping capabilities. We use the official implementation code: https://github.com/qiulin-w/safa.

UniFace:Face reenactment and swapping are two highly related tasks, but current research works almost always consider each task separately. In this work, the authors propose an efficient end-to-end unified framework to perform these two tasks. Unlike current methods that directly utilize pre-trained face structure prior networks to obtain facial attributes and identity information, the authors use a self-supervised approach to decouple and obtain attribute information represented as vectors and identity information represented as feature maps. The method does not rely on any facial priors during inference, making it perform better in extreme environments. Additionally, current methods do not fully consider the intrinsic connection between the two tasks, leading to reduced performance under a unified framework. The authors meticulously designed attribute and face transfer modules and efficiently combined corresponding modules according to the relationship between the two tasks to enhance the performance of each task. Experiments prove that the unified structure achieves more advanced results in both face reenactment and swapping tasks. We use the official implementation code: https://github.com/xc-csc101/UniFace.

MobileSwap:Advanced face swapping methods have achieved remarkable results. However, most of these methods involve many parameters and computations, making them challenging to apply in real-time applications or deploy on edge devices like smartphones. In this work, a lightweight Identity-aware Dynamic Network (IDN) is proposed that dynamically adjusts model parameters through identity information, for arbitrary face swapping. The provided IDN contains only 0.50M parameters and requires 0.33G FLOPs per frame, thus enabling real-time video face swapping on mobile devices. Furthermore, a stable training method based on knowledge distillation is introduced, and a loss reweighting module is adopted to achieve better comprehensive results. We use the official implementation code: https://github.com/Seanseattle/MobileFaceSwap.

e4s: e4s rethinks face swapping from the perspective of fine-grained face editing, termed "edit swapping." Moreover, e4s can handle facial occlusions with an average mask. The core is a novel Region GAN Inversion (RGI) method that explicitly disentangles shape and texture, allowing face swapping in the latent space of StyleGAN. Based on disentanglement, face swapping is reformulated as a simplified style and mask swapping problem. Recently, e4s has been improved, mainly addressing potential lighting discrepancies when transferring the source face's skin to the target image, ensuring that the swapped face maintains target lighting conditions while preserving the source skin. It also repairs post-swap faces, mainly potential mismatch areas that may occur during the mask swapping process, refining facial shapes. The latest generation speed is very slow, and it has lost continuity for video face swapping. We use the official implementation code: https://github.com/e4s2024/E4S2024.

FaceDancer:FaceDancer is a novel single-stage face swapping technique with real-time generative performance, utilizing Adaptive Feature Fusion Attention (AFFA) and Interpretative Feature Similarity Regularization (IFSR) modules. It achieves high-fidelity identity transfer without additional face segmentation processes while preserving target facial attributes such as expression, pose, and lighting. IFSR leverages a pre-trained identity encoder to preserve key facial attributes, while the AFFA module learns gated features for adaptive fusion of attributes and identity information. Experiments on various datasets show that FaceDancer outperforms existing methods in identity preservation and pose maintenance. It can reach 200ms on A100. We use the official implementation code: https://github.com/felixrosberg/FaceDancer.

AFS:AFS is a novel, high-fidelity, and straightforward face swapping method that explicitly decomposes the intermediate latent space W+ of a pre-trained StyleGAN into "identity" and "style" subspaces. In W+, a latent code is the sum of "identity" and "style" codes from the corresponding subspaces. Through this decomposition, face swapping is viewed as simple arithmetic operations in W+, namely, the sum of the source "identity" code and the target "style" code. This makes AFS more intuitive and elegant than other face swapping methods. We use the official implementation code: https://github.com/truongvu2000nd/afs.

FaceShifter:FaceShifter is a SOTA face swapping algorithm from 2020, mainly addressing the occlusion challenge encountered in face swapping. It proposes a two-stage framework for high-fidelity and occlusion-aware face swapping. AEI-Net achieves comprehensive integration of target attributes through an adaptive embedding integration network, addressing inconsistencies in lighting and shape; HEAR-Net handles occlusions in a self-supervised manner without manual annotations. Compared to existing methods, FaceShifter excels in fidelity and identity preservation. Not only is FaceShifter a powerful and practical tool, but it also provides an excellent platform for researchers and developers to explore face transformation technologies. We use the official implementation code: https://github.com/taotaonice/FaceShifter.

DiffSwap:DiffSwap is a 3D-aware conditional repair technique that uses a diffusion model for high-fidelity controllable face swapping. With identity features and facial landmarks guided conditional repair, it solves the shape retention problem inherent in traditional methods. The proposed midpoint estimation method requires only two steps to achieve identity constraints, improving the quality of the swap. The introduced Mask D is a technique to indicate which facial features need to be swapped. Mask D can identify and label different areas of the face, allowing for precise control over the face swapping process. Users can select specific areas to swap, such as eyes, nose, mouth, etc., achieving personalized face swapping effects. This technology has potential applications in many fields, such as movie special effects, video games, and virtual reality. Its reasoning speed is very slow. We use the official implementation code: https://github.com/wl-zhao/DiffSwap.

FOMM:FOMM is a pioneering work in reenactment algorithms, and many current algorithms are improvements based upon it, appearing in many public datasets. FOMM uses a self-supervised formula to separate appearance and motion information. To support complex movements, it employs a representation method consisting of a set of learned keypoints and their local affine transformations. The generator network models occlusions that occur during the target motion process and combines the appearance extracted from the source image with the motion extracted from the driving video. We use the official implementation code: https://github.com/AliaksandrSiarohin/first-order-model.

FS_vid2vid:vid2vid is a deep learning-based video generation method that trains a neural network model to transform input static images or a small number of video clips into continuous videos. Compared to traditional video generation methods, vid2vid has higher generation quality and consumes fewer computational resources. This makes it of great practical value in many application scenarios. FS_vid2vid is an improved method based on vid2vid technology. Its main idea is to synthesize new videos using a small number of video clips. This method greatly improves the diversity and flexibility of generated videos while maintaining high-quality generation, making it play a larger role in many application scenarios. We use the official implementation code: https://github.com/NVlabs/few-shot-vid2vid.

FADM:This paper is the first to use a diffusion model for face driving, proposing a facial animation framework based on an Attribute-Guided Diffusion Model (FADM). Previous GAN-based methods often produce unnatural distortions and artifacts due to complex motion deformations, while diffusion models have outstanding modeling capabilities. The main component is the Attribute-Guided Conditional Network (AGCN), which adaptively combines animation features and 3D facial reconstruction results, integrating appearance and motion conditions into the diffusion process, correcting unnatural artifacts and distortions, and enriching high-fidelity facial details through iterative diffusion refinement. We use the official implementation code: https://github.com/zengbohan0217/FADM.

FNeVR:Face reenactment still faces two major challenges: maintaining identity features and generating realistic images, mainly due to complex motion deformations and facial detail modeling. To solve these two problems, FNeVR designed a 3D Face Volume Rendering (FVR) module to enhance facial details in image rendering. The carefully designed architecture extracts 3D information, then introduces an orthogonal adaptive ray sampling module for efficient rendering, and designs a lightweight pose editor, allowing FNeVR to edit facial poses in a simple and effective way. We use the official implementation code: https://github.com/zengbohan0217/FNeVR.

TPSMM:TPSMM mainly solves the problem of large angles, i.e., cases where there is a significant gap in pose between the source and driving images. It proposes a thin-plate spline motion estimation method to generate more flexible optical flow, thus warping the feature map of the source image into the feature domain of the driving image. To more realistically restore missing areas, multi-resolution occlusion masks are used for more effective feature fusion. Additional auxiliary loss functions are designed to ensure clear division of labor among network modules and encourage the network to generate high-quality images. The method has been successfully applied to animating various objects such as faces, bodies, and cartoons, showing great potential in handling unseen operations and demonstrating strong generalization capabilities. We use the official implementation code: https://github.com/yoyo-nb/thin-plate-spline-motion-model.

LIA:The common method for face reenactment is to use structural representations extracted from driving videos (such as keypoints or regions), which is crucial for transferring motion from driving videos to static images. However, these methods may fail when there is a significant difference in appearance between the source image and the driving video. Additionally, extracting structural information requires extra modules, increasing the complexity of the animation model. LIA proposes using a self-supervised autoencoder to animate images without structural representations, mainly by simplifying the image animation process through linear navigation in latent space. The motion in the generated video is constructed by linear displacement in the latent space, learned by learning a set of orthogonal motion directions and using their linear combinations to represent any displacement in the latent space. LIA's idea is excellent, treating the motion between the source and driving images as a transformation in high-dimensional space, and performing arbitrary combinations in this high-dimensional space. We use the official implementation code: https://github.com/wyhsirius/LIA.

DaGAN:This work proposes a depth-aware generative adversarial network (GAN) that learns a depth estimation network in a self-supervised manner by training on facial videos without requiring additional 3D data as input. Thus, it can generate reliable facial depth maps for both source and driving images, producing higher quality talking head videos by capturing accurate facial 3D structures. The focus of this method is on combining 3D geometric information with deep learning to improve the quality and accuracy of generated talking head videos, especially when dealing with complex backgrounds and noise information. We use the official implementation code: https://github.com/harlanhong/cvpr2022-dagan.

LivePortrait:LivePortrait is the first performance-considerate face reenactment model, with a significant speed increase in generation compared to diffusion-based methods, reaching 12.8ms, suitable for industrial needs. The paper expands the training data to 69 million high-quality frames, adopts a hybrid image-video training strategy, upgrades the network architecture, designs better motion transformations and optimization objectives, and proposes compact implicit keypoints to effectively represent mixed shapes. It also designs splicing and two redirection modules to enhance controllability. We use the official implementation code: https://github.com/KwaiVGI/LivePortrait.

MCNet:MCNet, by the same author as DaGAN, is an improvement based on DaGAN. Intense and complex actions in the driving video cause blurring in the generated results because the static source image cannot provide sufficient appearance information for occluded areas or subtle facial expression changes, resulting in severe artifacts and significantly reducing generation quality. To solve this problem, a network module was designed to learn a unified spatial facial memory bank from all training samples, which can provide a rich facial structure and appearance priors to compensate for the generation of distorted source facial features. An effective query mechanism based on implicit identity representation is proposed, learned from discrete keypoints of source images. It greatly facilitates retrieving more relevant information from the repository for compensation. Testing shows that the more frontal the input single image of the face, the better the effect. Within a certain range of facial rotation angles, the synthesis of facial expressions and movements is very good. The larger the facial rotation angle, the greater the deformation. At the same time, the closer the face in the driving video is to the face in the target image (e.g., eye size, mouth size), the better. If the driving video has small eyes and the target image has large eyes, the synthesized target image may not close the eyes completely. We use the official implementation code: https://github.com/harlanhong/iccv2023-mcnet.

HyperReenact:Existing face reenactment methods are prone to visual artifacts, especially when there are significant head pose changes, or expensive fine-tuning is required to maintain source identity features. This paper leverages the realistic generative capabilities and decoupled attributes of the pre-trained StyleGAN2 generator. It first converts real images to their latent space and then uses a hypernetwork for source identity feature refinement and facial pose redirection, thus eliminating artifacts without external editing methods. The method operates under a one-time setup, allowing cross-subject reenactment without fine-tuning for specific subjects. But for unseen data, sometimes the decoupling effect is not very effective, and the generated video noise is relatively high. We use the official implementation code: https://github.com/stelabou/hyperreenact.

FRST: FRST is a current SOTA method that achieves cross-subject reenactment by transferring head movements and facial expressions from the driving video to the appearance of the source image. First, a Transformer-based encoder computes the latent representation of the source image. Then, using a Transformer-based decoder combined with keypoints and facial expression vectors from the driving frames, it predicts the output color of query pixels. It learns the latent representation of the source character in a self-supervised manner, separating its appearance, head pose, and facial expressions, suitable for cross-subject reenactment. It can be naturally extended to multiple source images, adapting to the facial dynamics of specific characters. We use the official implementation code: https://github.com/andrerochow/fsrt.

AniPortrait:AniPortrait is an audio- and reference-portrait-based animation generation framework that processes in two stages (Audio2Lmk and Lmk2Video), extracting 3D intermediate representations from audio and projecting them into a series of 2D facial keypoints. Then, using a robust diffusion model combined with a motion module, it transforms the keypoint sequence into realistic and temporally consistent portrait animations. Experimental results show that AniPortrait has advantages in facial naturalness, pose diversity, and visual quality, thus providing an enhanced perceptual experience. Additionally, it shows considerable potential in flexibility and controllability, effectively applicable in fields like facial action editing or face reenactment. However, it relies on high-quality 3D data and has GPU compatibility issues. We use the official implementation code: https://github.com/Zejun-Yang/AniPortrait.

MRFA:Existing methods mostly use a prior-based motion model (e.g., the local affine motion model or the local thin-plate-spline motion model), which can capture broad facial movements, but often produce artifacts in small movements in local areas (such as lips and eyes) because these methods have limited modeling capabilities for more refined facial movements. The authors designed a new unsupervised face animation method that learns both coarse and fine movements. First, it constructs a structure-related volume based on keypoint features of the source and driving images, then generates tiny facial movements from low to high resolution, and finally combines the learned motion refinement with coarse movements to generate a new image. We use the official implementation code: https://github.com/JialeTao/MRFA.

DPE:Head poses and facial expressions are always intertwined and transferred simultaneously in facial actions, which limits the direct application of this method in video portrait editing, as video portrait editing may only require modifying expressions while keeping the pose unchanged. One challenge in decoupling poses and expressions is the lack of paired data, such as data with the same pose but different expressions. Although some methods attempt explicit decoupling using 3D Morphable Models (3DMMs), due to the limited number of Blendshapes, 3DMMs are not accurate enough in capturing facial details, which affects the quality of action transfer. This paper introduces a novel self-supervised decoupling framework that can decouple poses and expressions without the need for 3DMMs and paired data. The framework consists of a facial encoding module, a pose generator, and an expression generator. The facial encoding module projects the face into a latent space, where the pose actions and expression actions can be decoupled, and pose or expression transfers can be conveniently performed in the latent space through addition. The two generators then render the modified latent codes into images. Furthermore, to ensure decoupling, this paper proposes a bidirectional cycle training strategy with carefully designed constraints. We use the official implementation code: https://github.com/Carlyx/DPE.

OneShot:This paper proposes a neural talking head video synthesis model for video conferencing. The model learns to synthesize talking head videos using a source image containing the target person's appearance and a driving video that determines the output motion. The paper encodes motion based on a novel keypoint representation, where specific identity and motion-related information are decomposed in an unsupervised manner. Extensive experiments demonstrate that the model outperforms competing methods on benchmark datasets. The compact keypoint representation enables video conferencing systems to achieve commercial H.264 standard visual quality while only using a tenth of the bandwidth, as the entire video does not need to be transmitted over the network; only the first frame image and subsequent 3D keypoint representations for each frame are required. Additionally, the keypoint representation allows users to rotate the head during the synthesis process, which is useful for simulating a face-to-face video conferencing experience. We use the official implementation code: https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis.

MakeItTalk:MakeItTalk brings faces or cartoon characters to life. It first extracts the facial landmarks from the image using existing methods, then extracts the content information and speaker information from the input speech, and uses these two features to obtain the corresponding changes in facial landmarks for the lips, head, and facial expressions. Finally, it combines these with the original image to produce a video. The main advantage is the separation of content information and speaker information in the speech, which allows for better lip-sync and head movements and facial expressions that are more consistent with the speaker themselves. We use the official implementation code: https://github.com/yzhou359/MakeItTalk.

Portrait4D:Existing one-shot 4D head synthesis methods typically rely on monocular videos and learn with the aid of 3DMM reconstruction, which is highly challenging and limits their ability to perform reasonable 4D head synthesis. This paper proposes a method to learn one-shot 4D head synthesis through large-scale synthetic data. The key is to first learn a part-based 4D generative model from monocular images through adversarial learning to synthesize multi-view images with different identities and full motion as training data; then, using a transformer-based animatable tri-plane reconstructor, the method learns 4D head reconstruction using synthetic data. A novel learning strategy is implemented to enhance the generalization ability to real images by disentangling the learning processes of 3D reconstruction and reenactment. However, the generated video characters have unclear tooth jitter, and for blinking actions, there may be erroneous enlargement of the face. We use the official implementation code: https://yudeng.github.io/Portrait4D.

SadTalker:Current methods for generating speaking persons have problems such as unnatural head movements, distorted facial expressions, and identity modifications. These issues mainly stem from the learning of coupled 2D motion fields, while the explicit use of 3D information also leads to stiff expressions and incoherent videos. Therefore, this paper proposes SadTalker, which generates 3D motion coefficients (head pose, expressions) from audio using a new 3D facial renderer to generate head movements. To learn authentic motion coefficients, the researchers model the connection between audio and different types of motion coefficients separately, learning accurate facial expressions from audio through distilled coefficients and 3D rendered faces; a conditional VAE, PoseVAE, is designed to synthesize different styles of head movements. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of facial rendering and combined to synthesize the final video. Experiments show that this method achieves state-of-the-art performance in motion synchronization and video quality. However, the issue of unclear teeth remains unresolved. We use the official implementation code: https://github.com/winfredy/sadtalker.

EchoMimic:Previous methods only used audio or facial keypoints to drive images into videos, and while they could produce satisfactory results, certain issues persisted. For example, methods driven solely by audio might sometimes be unstable due to the relatively weak audio signal, while methods driven solely by facial keypoints, although more stable in driving, could lead to unnatural results due to over-control of facial keypoint information. EchoMimic uses both audio and facial landmarks for training. By implementing a novel training strategy, EchoMimic can generate portrait videos not only from audio and facial landmarks separately but also from a combination of audio and selected facial landmarks. We use the official implementation code: https://github.com/BadToBest/EchoMimic.

Audio2Head:Previously, neural network-based algorithms enabled the lip movements in facial videos to match speech relatively well, but issues remained, such as unnatural head movements, incoherent videos, and numerous artifacts. To address these issues, Audio2Head models head movements separately, proposing a spatially encoded neural network for the natural prediction of head movement sequences. To model the motion of the entire image related to speech, Audio2Head uses a speech precursor to generate a dense motion field for the entire image, which then guides image synthesis. This technology can quickly respond to audio input, achieving real-time head movement generation, and only requires a single image of a person to adapt to various audio inputs. We use the official implementation code: https://github.com/wangsuzhen/Audio2Head.

VideoReTalking:The lips generated by current generic talking person models are blurry, and they do not support emotional editing. Inspired by wav2lip, the authors propose a new model to drive lip synthesis with audio. In wav2lip, two sets of five frames each are required as input: one set is the GroundTruth with the lower half masked, and the other set is five random frames from the original video (different from the GroundTruth). The masked image frames are obviously the images for which we need to generate lips with audio, while the random image frames provide pose references for lip generation. The authors note that the model is highly sensitive to the pose reference image frames, as the lip information contained in these frames leaks to the model as prior knowledge. If random image frames are used directly as pose references, the generated images often result in asynchronous outcomes. Therefore, the authors modify the random image frames used as pose references by neutralizing facial expressions before inputting them into the model as pose references. Following this idea, modified facial expressions of happiness or sadness can also be used as pose references, naturally generating talking videos with corresponding emotions. VideoReTalking has high practical value, such as silent processing of videos when silent videos are lacking. We use the official implementation code: https://github.com/vinthony/video-retalking.

Hallo:This paper is based on an SD scheme that can generate visually appealing, temporally coherent, and complex facial animation audio-driven videos. It is an end-to-end solution, where the authors introduce a hierarchical audio-driven visual synthesis module that uses a cross-attention mechanism to establish correspondences between audio and visual features (lips, expressions, and poses) to improve the alignment between audio input and visual output, including lip movements, expressions, and postural actions. The proposed hierarchical audio-driven visual synthesis provides adaptive control over the diversity of expressions and postures, achieving more effective personalized customization to accommodate different identities. The training also uses some tracks to enhance the effects. We use the official implementation code: https://github.com/fudan-generative-vision/hallo.

Real3D-Portrait:In recent years, with the continuous advancement of single-image-driven virtual human technology, lip accuracy and image quality have been steadily improving; Real3D-Portrait for the first time achieves single-image-driven virtual human video synthesis supporting large-scale pose movements with advanced single-image 3D reconstruction technology, further unlocking the motion freedom of single-image-driven virtual humans. Its feature of reconstructing 3D avatars also gives it the potential to be applied in spatial visual products. It is foreseeable that, with the continuous iteration and popularization of technology, virtual humans will appear in various application scenarios such as intelligent assistants, virtual reality, and video conferencing. With Real3D-Portrait, single-image-driven virtual human algorithms are expected to make speakers 'move' more realistically in 2D/3D scenes. However, Real3D-Portrait is not without flaws at this stage; possibly due to a small data volume and sample quality issues, the model sometimes struggles to produce clear and accurate results for areas obscured in the input image, such as teeth and side faces. We use the official implementation code: https://github.com/yerfor/Real3DPortrait.

Wav2Lip:Wav2Lip is a classic talking person model, its greatest contribution being the provision of a lip-sync discriminator, which offers new stringent evaluation benchmarks and metrics to accurately measure lip-sync in unrestricted videos. Many products are currently using wav2Lip as their underlying algorithm, enhancing facial and dental clarity by adding super-resolution models (such as GFPGAN). We use the official implementation code: https://github.com/Rudrabha/Wav2Lip.

Controlnet:Controlnet is a popular explicit conditional control method currently in widespread use. ControlNet is a neural network architecture designed to add spatial conditional control to large, pre-trained text-to-image diffusion models. It locks down large diffusion models that are ready for production and reuses these models' deeply and robustly encoded layers, trained on billions of images, as a powerful backbone for learning diverse conditional controls. We use the official implementation code: https://github.com/lllyasviel/controlnet.

DeltaEdit:Previous editing methods had one flaw: for different text prompts, they had to undergo different optimization processes, which was inflexible during training or inference and did not generalize well to any other unseen text. The authors believe that the key to mitigating this problem is to accurately establish the relationship between the text feature space and the StyleGAN latent space within one model. Therefore, a method for image editing in the StyleGAN space is proposed, conditioned on the corresponding embeddings in the CLIP image space, without any text supervision. This method involves randomly selecting two images from a training image dataset, extracting their CLIP image embeddings and the pre-trained StyleGAN model to extract latent codes in the S space. Then, the extracted latent codes are used to predict the manipulation direction. The paper also proposes a solution that directly uses image embeddings to construct pseudo-text conditions based on the aligned features of the joint CLIP image-text space. We use the official implementation code: https://github.com/yueming6568/deltaedit.

Face2Diffusion:Previous diffusion-based editing methods aimed to insert specific facial features from images into pre-trained text-to-image diffusion models. However, past methods still faced challenges in maintaining identity similarity and editability, as they were overfitted to training samples. This paper proposes a Face2Diffusion (F2D) method for high editability facial personalization. The core idea behind F2D is to remove identity-irrelevant information from the training process to prevent overfitting issues and enhance the editability of encoded faces. F2D includes three novel components: first, a multi-scale identity encoder provides well-separated identity features while maintaining the benefits of multi-scale information, thus improving camera pose diversity; second, expression guidance separates facial expressions from identity, enhancing controllability of facial expressions; third, category-guided denoising regularization encourages the model to learn how to denoise faces, thereby improving the text alignment of the background. We use the official implementation code: https://github.com/mapooon/face2diffusion.

PhotoMaker:Recently, text-to-image generation technology has made significant progress in synthesizing realistic human photos that can be conditionally generated based on given text prompts. However, existing personalized generation methods cannot simultaneously meet the requirements of high efficiency, high identity (ID) fidelity, and flexible text controllability. This paper is an efficient personalized text-to-image generation method, which primarily encodes any number of input ID images into a stacked ID embedding to preserve ID information. This embedding, as a unified ID representation, not only encapsulates the features of the same input ID comprehensively but also accommodates the features of different IDs for subsequent integration. This paves the way for more interesting and practically valuable applications. Moreover, to drive the training of PhotoMaker, we propose an ID-oriented data construction pipeline to assemble training data. Nourished by the dataset constructed through the proposed pipeline, our PhotoMaker outperforms test-time fine-tuning methods in ID preservation capability, while also providing significant speed boosts, high-quality generation results, powerful generalization capabilities, and a wide range of applications. We use the official implementation code: https://github.com/TencentARC/PhotoMaker.