Self-supervised Transformer for Deepfake Detection (2024)

2.1 Deepfake Detection

Since the deepfake technique has caused severe societal concerns, many effective countermeasures [28, 34, 22, 1, 45, 25, 18, 42, 6] have been proposed against it. According to the feature extraction types, current methods can be roughly categorized into two types: textural feature based methods and semantic feature based methods. As the name implies, textural feature based methods focus on capturing low-level textural information from different domains. Two-branch[28] introduced an extra CNN stem with deep Laplacian-of-Gaussian filter in the CNN-RNN architecture for acting as a band-pass filter to amplify artifacts. Patch-forensics[4] proposed a classifier with limited receptive fields that focus on textures in small patches to captured local errors. Multi-attention [45] attempt to introduce the multi-regional attention mechanism into deepfake detection, which is inspired by fine-grained classification.

To boost the generalization ability, Face X-ray [22] and Patch-wise Consistency Learning [46] leverage the clues of image blending between the background and the altered face region. These methods achieve impressive performances in cross-data evaluation. However, they are easily influenced by video compression and noise disturbance since the boundary clues are highly fragile to post-processing.

Recently, researchers find that high-level semantic features show excellent robustness in dealing with both cross-data tests and post-processing operations. Lipforensics [18] extract the representations of lip movement by using a pre-trained network. The lip feature extractor is pre-trained in a supervised manner on Lipreading in the Wild(LRW)[10] dataset, which is commonly used for the lipreading tasks. The network takes a multi-branch temporal convolution network(MSTCN)[16] as backend and fine-tuned on deepfake datasets. It achieves state-of-the-art generalization ability. However, to pre-train such a robust lip feature extractor requires a large-scale well-annotated dataset, which is extremely costly. In this paper, we propose a self-supervised transformer for pre-training, in this way, we can significantly reduce the annotation cost for pre-training data and obtain good scalability.

2.2 Contrastive Learning

Recently, general pre-trained models have been widely used for fine-tuning on downstream tasks. Self-supervised pre-training attracts tremendous attention for its versatility. Self-supervised learning enables us to learn meaningful representation for various classification tasks without relying on labeled data. Contrastive learning is a popular self-supervised strategy that encourages the features of the same instances to be close while features of different instances to be distant. Generally, we can take different views of the same sample data as positive pairs and views from unmatched data as negative pairs to construct a classification task for discriminating instances. For pre-training semantical image representations, [7, 8] obtain different views of the same image by random augmentations. For pre-training video classification models[17], positive pairs are fragments from the same video while negative pairs are fragments from different videos. Besides the uni-modal contrastive learning, natural multi-media data have cross-modal consistency.

Many previous self-supervised learning methods are designed in the cross-modal predictive way, while it is also suitable for using contrastive learning that takes each modal as different views with shared information. For example, Ommer et al.[38] proposed contrastive learning for the frame and optical flow in videos. [35] demonstrate a simple pre-training task, that is, predicting which caption goes with which image is an efficient and scalable way to learn good image representations. Some recent works[48, 29, 27] leverage correspondence between audio and visual signals for learning semantic representations, they achieve strong transfer learning performance on downstream tasks like action recognition and lipreading. Inspired by previous successful works, in this paper, we try to learn a general representation of lip movement by audio-visual consistency using contrastive learning.

3.1 Overview

Most deepfake generation methods synthesize fake faces frame by frame without considering motion coherence, especially in the lip regions where movements are most frequent and complex while talking. Thus the irregularity of lip movement can be a common feature of deepfake video and such a high-level semantic feature is robust to various video post-processing. To learn semantical representations of lip movements, we apply cross-modal contrastive learning by encoding utterance features and lip motion features from videos into a common space. The we use instance discrimination learning to enforce corresponding audio and visual features matching each other, which shares a similar spirit with the lipreading task. The learned lip representations can transfer to expose the irregularities in deepfake videos. Benefit from the primitive features of lip movements, the video encoder would be less prone to overfitting non-transferable artifacts when fine-tuning on deepfake detection task.

3.2 Audio-Visual Contrastive Learning pre-train

For cross-modal contrastive learning, our model consists of a video encoder and an audio encoder. In the pre-train stage, each encoder uses an additional MLP head for projecting the raw features into the common space.We take a pre-trained wav2vec2 model[2] for speech recognition as the backbone of the audio encoder. The wave2vec2 model encodes raw audio into a sequence of local wave features by one dimensional CNN layers which are frozen during training and then inference the features by transformer layers to get semantical audio representation array. We apply temporal adaptive average pooling for converting the output feature array from the last transformer layer to a fixed length token sequence, then the sequence is reshaped as the input of the MLP projection head.

We design a spatial-temporal framework as shown in figure Figure1 for encoding video features. It has two stages, in the frontend stage, a 3D convolutional layer with kernel size [5,7,7] extracts the optical flows from raw videos, then a following 2D ResNet module encodes the local lip movement feature from the optical flows. In the backend stage, we convert the local lip movement feature of each frame into a feature sequence with linear projection, then we use a temporal transformer to get the global semantic video representation. Just like the audio encoder, we also apply temporal adaptive average pooling and MLP projection head for encoding the final video feature.

Previous contrastive learning methods for video representations [29, 27, 12, 32] usually leverage 3D CNN architectures for general video understanding. Compared to 3D CNN, our architecture separates the process of encoding local lip movement feature and global semantic video representation. Our framework makes the fine-tuning process easier because we can freeze the frontend for keeping the low-level feature representations learned during the pre-training, only fine-tune the backend for capturing the long-term inconsistency of fake videos.

In our framework, we use the transformer as the backend of the encoder. Transformer has succeeded in many NLP and vision tasks since it can capture long-term dependency and preserve less inductive bias than CNN. Thus we believe it is more suitable for large-scale pre-trained models. The shape of output tensor from frontend is noted as $T\in{B\times L\times N}$, $B$ is the batch size, $L$ is a static sequence length with padding, $N$ is the dimension of pooled spatial feature, for example, $N$ is 512 for ResNet18 frontend. We use a linear projection for transforming $T$ to the input features of the transformer, and we add learnable positional embeddings to input features for introducing time relationships. In the experiment part, we validate the model with transformers backend can achieve better performance than those with general CNNs.

We utilize the intrinsic correlation of lip movement and speech audio to learn generic representations of them. Following the contrastive learning manner, we use the InfoNCE[41] loss to distinguish positive audio-visual feature pairs from a bunch of negative pairs. Video encoder $E_{v}$ encodes a video clip into video embedding $m_{v}$ and similarly audio encoder $E_{a}$ encodes an audio clip into audio embedding $m_{a}$. Then $m_{v}$ and $m_{a}$ are projected to common space by MLP heads as $z_{v}$ and $z_{a}$. The loss is shown in equation Equation3, we respectively take the video features and the audio features as query and the other modal as keys to get loss $\mathcal{L}_{va}$ and $\mathcal{L}_{av}$, then we simply use their average as the contrastive loss.

Here the label ⁺ means positive sample pairs and ^- means negative sample pairs, $\tau$ is the temperature which is set 0.1. For each iteration in the pre-train stage, we sample a mini-batch of video clips, extract segments of synchronized frames and waves with random offset for each video clip as positive audio-visual pairs while all the unrelated items from the mini-batch act as negative pairs.

3.3 Fine-tuning on Deepfake Dataset

To mitigate the influence of different frame rates, we convert all videos to 25 fps and all audios to a 16kHz sample rate. Then we trace the face of the speaking person with RetinaFace [13] and SyncNet [11]. We crop the selected faces with a tight boundary and use FAN [3] to get the 68 landmarks. We align each face using 5-landmarks and resize the face images to 256x256. We then crop a 96x96 region at the center of the lip landmarks region. After preprocessing, a video’s lip regions can be represented as $F_{v}\in R^{[T,H,W]}$, where $T$ is the length in time and $H,W$ are height and width. In the pre-train stage, we randomly sample a clip with fixed length $L_{p}$ and corresponding raw audio with length $640L_{p}$ from the video. We apply identical image augmentations on each frame including random crop to 88x88, motion blur, Gaussian noise and random brightness.

The computation complexity of self-attention is the square of the sequence length that forbids us to predict with a very long video sequence. For efficiency, we chunk videos into clips with a smaller length $L_{f}$ and use the same augmentations as it in the pre-train stage for fine-tunning. We use 2-second clips of videos for pre-training and only 1-second clips of videos for deepfake detection, the longer sequence in contrastive learning can diminish inner-modal similarity to reduce impacts of false-negative pairs and a shorter sequence can speed up the inference procedure. It is worth mentioning that the local lip movement feature extracted by the frontend is irrelevant to the sequence length.

4.1 Implement Details

We use $VoxCeleb2$ dataset [9] and part of $AVSpeech$ dataset [15] for pre-training, totally contains 2,800,000 speech video clips. Each video clip includes more than 100 continuous frames with a trackable face. We use AdamW optimizer for pre-train, the initial learning rate is 0.01 and decay 0.9 for each epoch, the minimum learning rate is 0.0001. The hyper-parameter of each transformer layer is 1024 dims, 8 attention heads, 128 dims for each attention head, 2048 intermediate dims and 0.2 dropout rate. The dimension of features for contrastive learning is 256. We end the pre-training when the training loss stops descending for 3 epochs. We conduct extensive experiments with different hyper-parameter combinations to determine the optimal selection for our method. Parts of the results are listed in Table4. Finally, we choose ResNet18 as frontend and 6 layers transformer as backend for the balance of efficiency and accuracy.

We fine-tune our pre-trained video encoder on original FaceForensics++ dataset[36] (FF++, 4 generation methods: Deepfake, FaceSwap, Neuraltextures, Face2Face and Original videos, 720 videos for each class). And we use the test set of Celeb-DF v2 [24] (518 videos), Deeper-forensics [19] (140 videos, paired with original videos of FF++ ), 140 Faceshifter videos from FF++ (paired with original videos) and DFDC [14] (3000 videos selected from the test set, exclude extremely corrupted videos) for evaluating the cross dataset ability. Following the commonly used evaluation metrics in previous deepfake detection works, we leverage video level ROC-AUC score and binary classification accuracy for the evaluation of the detection performances.

Notably, due to the training set being unbalanced (the fake videos are 4 times as many as real videos), previous methods usually over-sample real videos. In this work, we sample real videos with similar quality from the pre-train dataset to make the training set balanced.

4.2 Evaluation on FaceForensics++

FaceForensics++ is the most widely used dataset for deepfake detection. The original videos of the FF++ dataset include speeches by people of different races. The dataset chooses similar identity pairs for face-swapping, a total of five face swapping methods are used to generate face forgeries on each pair. To consist with previous works, we only use 4 face forgery methods: Deepfake, FaceSwap, Neuraltextures, Face2Face to fine-tune our models, and we treat the FaceShifter as a discrete dataset. There are three compression rates of the FF++ dataset, uncompressed (Raw), slightly compressed (HQ), and heavily compressed (LQ). The uncompressed dataset is relatively easy for spotting deepfake artifacts, but the challenge is increasing with compression rate, in the LQ videos, most textural features are lost which is extremely challenging for detection.

In Table1, we report the video-level AUC and accuracy for each compression rate compared to other methods. Among which, Xception [36], CNN-aug [43], Patch-based [4], Two-branch [28] and CNN-GRU [37] are networks without special pre-train strategies. Face X-ray [22] generates training data by blending real faces and being supervised with blending boundary regression. Lipforensics [18] is pre-trained in a supervised manner on Lipreading in the Wild (LRW)[10] dataset. To parallel compare our self-supervised method to the supervised one, we also use the same network structure and pre-train it on LRW with the same training strategy of Lipforensics in a supervised manner, the model is denoted as Lipforensics*. Apparently, our self-supervised pre-trained model outperforms the supervised one.

From the results in Table1, we can observe that our model pre-trained in self-supervised manner performs much better than it in supervised manner. Compared to the state-of-the-art Lipforensics method, our model also achieves comparable performances on raw videos and HQ videos while slightly dropping on LQ videos.

4.3 Evaluation of Cross-manipulation Ability

Generalizing to unseen forgery classes is challenging in deepfake detection because the artifact patterns are different in each manipulation methods. Some deepfake detection methods trained with several forgery classes may overfit multiple class-specific artifacts but not capture the common feature transferable to new method.

In this experiment, we evaluate the models’ generalizaiton ability cross manipulation methods with the leave-one-out strategy. Each time, we train the model with 3 forgery classes in FF++ HQ dataset and test with the remaining forgery class. The results are shown in Table2. Our method achieves better performance than most baseline methods, this indicates the lip movement features captured by our model are well generalizable.

4.4 Evaluaton of Cross-datasets Generalization

In real deepfake detection scenarios, the distribution of deepfakes is more complex. The forged videos are not only diverse in source videos and generation methods, there is also a diversity in post-processing methods. Thus the domain generalization ability is a significant metric for deepfake detection models. Since there are large domain gaps between different deepfake datasets, in this part, we evaluate the domain generalization ability of deepfake detectors by cross-dataset test. We fine-tune our model on the FF++ training set with four manipulation methods (DF, F2F, FS, NT), and report the AUC scores tested on Celeb-DF, DFDC, FaceShifter and DeeperForensics dataset, respectively.

In Table3, our method achieves state-of-the-art performances in generalization to Celeb-DF-v2, DFDC and FF++ FaceShifter dataset. Our self-supervised pre-trained model surpasses Lipforensics and the same model using supervised training strategy ( Lipforensics*) for an average AUC of 0.7 percent. The performances on Facesh*ter and Deeperforensics are obviously better than those on Celeb-DF-v2 and DFDC. It is probably because these two datasets share the same original videos with the FF++.

4.5 Ablation Study

4.5.2 Effect of Pre-training Dataset Scale

Compared to the supervised manner, self-supervised learning can leverage a larger amount of training data. To be specific, the labeled dataset LRW for lipreading task contains about 500,000 video clips of 500 words with precise boundary annotation. It only covers a limited part of real speech scenes but costs expensive human efforts in annotating. Contrastly, self-supervised learning only requires natural speech videos, which are very easy to collect. Empowered by self-supervised pre-train, we can improve the deepfake detection accuracy without requiring extra fake videos as training data. In this part, we give a simple investigation of how the quantity of unlabeled real video for pre-train affects the model’s accuracy and the cross-dataset ability. We draw the relation curves in Figure2 and Figure3. The blue lines are ResNet18 based models and the red lines are ResNet50 based models, we also plot the AUC of the model pre-trained on the LRW dataset with horizontal dotted lines. The abscissa of 0 represents that the model has not been pre-trained.

The curves in Figure2 and Figure3 validate that the models pre-trained with more original videos could improve both in-dataset and cross-datasets accuracies. When pre-trained with 480,000 video clips, the performance exceeds the same network pre-trained with 500,000 labeled video clips from LRW dataset. And the performance can still improve with the increase in training videos. From the comparison between the ResNet18 based model and the ResNet50 based model, we observe that without pre-train, the larger model does not outperform the smaller model on FF++ HQ while achieving only minor ascendency for generalization. But the models obtain obvious gains from pre-training. The AUC of the ResNet50 based model grows faster and saturates slower than ResNet18 due to the larger model scale. In conclusion, we demonstrate that larger-scale pre-training helps boost the models’ performances. This highlights another advantage of our self-supervised method, that is, significantly reducing the data annotation costs to provide a larger scale of training data. However, larger-scale training data inevitably leads to higher computing resources demands, which might be a limitation.

4.5.3 Robustness of Pre-train Features

As aforementioned, the video in real-world scenarios may be accompanied by various disturbances, which requires that the detection model should be robust to various degradations of video quality. In our framework, we use massive real videos with various definitions to pre-train the video encoder. In this way, the learned representations of lip movement would be more robust to corruption. When fine-tuned for deepfake detection, we freeze the frontend of the video encoder so that the perturbations in videos would be reduced in the intermediate features. To evident the robustness of our model, we visualize the features of clean videos and corrupted videos extracted from the frontend and backend by our model.

We visualize the features from the self-supervised pre-trained model and the train-from-scratch model in Figure5. The models are trained with FF++ Raw videos. The left column shows t-SNE embedded features from the Raw test set and the right column shows features from the HQ test set(compressed with H264 codec) transformed with Raw t-SNE embeddings. Row (a) and row (c) illustrate frontend and backend features of the non-pre-trained model separately, row (b) and row(d) are corresponding features of the self-supervised pre-trained model. We eventually observe that the distribution of the features trained with audio-visual contrastive learning changes more slightly compared to the train-from-scratch one. As a consequence, we believe that the deepfake detection models with self-supervised pretrain are less susceptible to interference from video compression.

Further, we investigate the robustness of our methods to various video corruptions. We take four common corruption types to simulate various degradations of definition: adding block-wise distortions, Gaussian blurring, JPEG compression, and H264 video compression, each corruption with five severity levels (details refer to [19]). In Figure4, we give the AUC curves under different corruption types. In subjects of Block-wise distortion and video compression, our self-supervised method performs better robustness than supervised pre-train, and in other two subjects, our method also achieves comparable performances.