第一次竞赛记录

阅读量

背景

竞赛主题

SemEval 2023 - Task 10 - Explainable Detection of Online Sexism (EDOS)

数据集

  • 有标签数据

    • 标签分布
      • image-20221123100717895
    • 标签模式
      • image-20221123100825597
      • image-20221123100835504

  • 无标签数据

    • image-20221123101136708

小分析

  • 有标签数据足以用来做微调

  • 与有标签数据同属于一个社区的无标签的数据需要利用起来

    • 受语言模型处理领域内的性能限制，我们可以充分领域内的数据继续预训练语言模型
    • 参考PET论文，我们可以通过大型语言模型给无标签打上软标签

  • 由于该任务类别不均衡，正负例比例为1：3

    • loss优化：平衡交叉熵、focal loss（见下一章的第二小节代码）
    • 采样优化：分层采样
    • 样本均衡：数据增强

我尝试的trick

🎈情感prompt

  • 带有性别歧视的样本情感极向一般偏负
    • 性别歧视 -> negative
    • 非性别歧视 -> 用已在情感分类任务中fine-tune的PLM预测情感极向

🎈社交媒体数据清理

  • Emoji替换成自然语言。如：😀 -> smile
  • √ 去除Emoji
  • √ 去除特定的标点符号
  • √ 去除url
  • √ 去除html
  • 全转化成小写
  • √ 缩写词展开。如：AKA -> “Also Known As”

🎈Focal Loss function

  • what

    • 主要关注解决样本不均衡与难样本问题?

  • why

    • CE

    • focal loss

  • 实现代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import torch
import torch.nn as nn
import torch.nn.functional as F


class FocalLoss(nn.Module):
    """Multi-class Focal loss implementation.
    Args:
        gamma (float): The larger the gamma, the smaller
            the loss weight of easier samples.
        weight (float): A manual rescaling weight given to each
            class.
        ignore_index (int): Specifies a target value that is ignored
            and does not contribute to the input gradient.
    """

    def __init__(self, gamma=2, weight=None, ignore_index=-100):
        super().__init__()
        self.gamma = gamma
        self.weight = weight
        self.ignore_index = ignore_index

    def forward(self, input, target):
        logit = F.log_softmax(input, dim=1)
        pt = torch.exp(logit)
        logit = (1 - pt)**self.gamma * logit
        loss = F.nll_loss(
            logit, target, self.weight, ignore_index=self.ignore_index)
        return loss

🎈对抗训练FGM

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class FGM(object):
    def __init__(self, model, emb_name, epsilon=1.0):
        # emb_name这个参数要换成你模型中embedding的参数名
        self.model = model
        self.epsilon = epsilon
        self.emb_name = emb_name
        self.backup = {}

    def attack(self):
        for name, param in self.model.named_parameters():
            if param.requires_grad and self.emb_name in name:
                self.backup[name] = param.data.clone()
                norm = torch.norm(param.grad)
                if norm != 0 and not torch.isnan(norm):
                    r_at = self.epsilon * param.grad / norm
                    param.data.add_(r_at)

    def restore(self):
        for name, param in self.model.named_parameters():
            if param.requires_grad and self.emb_name in name:
                assert name in self.backup
                param.data = self.backup[name]
        self.backup = {}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
fgm = FGM(model, epsilon=1, emb_name='word_embeddings.')
    for batch, (x, y) in enumerate(dataloader, start=1):
        optimizer.zero_grad()
        x = x.to(device)
        y = y.to(device)
        
        # 正常训练
        y_hat = model(x)
        loss = loss_fn(y_hat, y)  
    
        # 反向传播，得到正常的grad
        loss.backward()
        
        # 对抗训练
        fgm.attack() # 在embedding上添加对抗扰动
        y_attack_hat = model(x)
        loss_adv = loss_fn(y_attack_hat, y)
        
        # 反向传播，并在正常的grad基础上，累加对抗训练的梯度
        loss_adv.backward() 
        fgm.restore() # 恢复embedding参数
        
        # 更新参数
        optimizer.step()
        scheduler.step()
        total_loss += loss.item()

🎈R-Drop

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
from balanced_loss import Loss
from torch import nn
from torch.nn import functional as F

# 使用R-drop时采用的loss
class RDropLoss(nn.Module):
    """
    R-Drop for classification tasks.
    Example:
        criterion = RDrop()
        logits1 = model(input)  # model: a classification model instance. input: the input data
        logits2 = model(input)
        loss = criterion(logits1, logits2, target)     # target: the target labels. len(loss_) == batch size
    Notes: The model must contains `dropout`. The model predicts twice with the same input, and outputs logits1 and logits2.
    """
    def __init__(self):
        super(RDropLoss, self).__init__()
        # 交叉熵损失
#         self.ce = nn.CrossEntropyLoss()
		# balanced focal loss
        self.ce = Loss(
            loss_type="focal_loss",
            samples_per_class=samples_per_class,
            class_balanced=True
        )
#         self.ce = FocalLoss()
        
        # KL散度损失
        self.kld = nn.KLDivLoss(reduction='none')

    def forward(self, logits1, logits2, target, kl_weight=4.):
        """
        Args:
            logits1: One output of the classification model.
            logits2: Another output of the classification model.
            target: The target labels.
            kl_weight: The weight for `kl_loss`. The range of this value is [1~5], 4 be selected by original paper

        Returns:
            loss: Losses with the size of the batch size.
        """
        ce_loss = (self.ce(logits1, target)+ self.ce(logits2, target)) / 2
        kl_loss1 = self.kld(F.log_softmax(logits1, dim=-1), F.softmax(logits2, dim=-1)).sum(-1)
        kl_loss2 = self.kld(F.log_softmax(logits2, dim=-1), F.softmax(logits1, dim=-1)).sum(-1)
        kl_loss = (kl_loss1.mean() + kl_loss2.mean()) / 2
        loss = ce_loss + kl_weight * kl_loss
        
        return loss
    
loss_fn = RDropLoss()

🎈EMA

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
class EMA():
    def __init__(self, model, decay):
        self.model = model
        self.decay = decay
        self.shadow = {}
        self.backup = {}

    def register(self):
        for name, param in self.model.named_parameters():
            if param.requires_grad:
                self.shadow[name] = param.data.clone()

    def update(self):
        for name, param in self.model.named_parameters():
            if param.requires_grad:
                assert name in self.shadow
                new_average = (1.0 - self.decay) * param.data + self.decay * self.shadow[name]
                self.shadow[name] = new_average.clone()

    def apply_shadow(self):
        for name, param in self.model.named_parameters():
            if param.requires_grad:
                assert name in self.shadow
                self.backup[name] = param.data
                param.data = self.shadow[name]

    def restore(self):
        for name, param in self.model.named_parameters():
            if param.requires_grad:
                assert name in self.backup
                param.data = self.backup[name]
        self.backup = {}

# ===================== Example =====================
# 初始化
# ema = EMA(model, 0.999)
# ema.register()

# # 训练过程中，更新完参数后，同步update shadow weights
# def train():
#     optimizer.step()
#     ema.update()

# # eval前，apply shadow weights；eval之后，恢复原来模型的参数
# def evaluate():
#     ema.apply_shadow()
#     # evaluate
#     ema.restore()

🎈Attention Pooling

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class AttentionPooling(nn.Module):
    def __init__(self, in_dim):
        super().__init__()
        self.attention = nn.Sequential(
            nn.Linear(in_dim, in_dim),
            nn.LayerNorm(in_dim),
            nn.GELU(),
            nn.Linear(in_dim, 1),
        )

    def forward(self, last_hidden_state, attention_mask):
        w = self.attention(last_hidden_state).float()
        w[attention_mask==0]=float('-inf')
        w = torch.softmax(w,1)
        attention_embeddings = torch.sum(w * last_hidden_state, dim=1)
        return attention_embeddings

🎈超参数搜索

  • 搜索方法

  • 一些固定超参数

    • seed: 固定42不搜索
    • lr_scheduler_type: 固定为liner
    • warmup_ratio: 固定为0.2

🎈领域内MLM

Ps:因为社交媒体领域非常特殊，表达和写作形式不是一般领域的典型，因此，在一般领域语料库上训练的模型可能没有像特定领域模型那样接触到这类文本。

🎈k折分层交叉验证 & 集成

🎈数据增强

  • 回译

  • 替换同义词

  • 工具：nlpaug

竞赛验证阶段

第一阶段，确认不同任务的模型和超参数

我手动对14000条样本按照9: 1切分成样本均衡的训练集和验证集，在三个子任务的验证集上确定模型的超参数。

subtask-a

模型 lr / batchsize epoch seed dropout scheduler
HPL/roberta-large-unlabeled-labeled-gab-reddit-task-semeval2023-t10-150000sample 9e-6 / 16 6 42 0.3 linear 0.2
bertweet-large 1e-5 / 16 6 42 0.3 linear 0.2

subtask-b

模型 lr / batchsize epoch seed dropout scheduler
bertweet-large 2e-5 / 16 8 42 0.3 linear 0.2

subtask-c

模型 lr / batchsize epoch seed dropout scheduler
bertweet-large 2e-5 / 16 8 42 0.3 linear 0.2

##第二阶段，尝试不同的trick

subtask-a

模型 trick macro-f1 exp-no
HPL/roberta-large-unlabeled-labeled-gab-reddit-task-semeval2023-t10-150000sample balance focal loss + FGM 0.859 76
HPL/roberta-large-unlabeled-labeled-gab-reddit-task-semeval2023-t10-150000sample balance focal loss + RDrop(α=4) 0.8631 83
HPL/roberta-large-unlabeled-labeled-gab-reddit-task-semeval2023-t10-150000sample balance focal loss + RDrop(α=4) + FGM 0.8683 87
HPL/roberta-large-unlabeled-labeled-gab-reddit-task-semeval2023-t10-150000sample balance focal loss + RDrop(α=4) + FGM + EMA 0.8681 94
bertweet-large balance focal loss + RDrop(α=4) + FGM + attention pooling 0.867911 142
bertweet-large balance focal loss + RDrop(α=4) + FGM + attention pooling + EMA 0.86610 144

subtask-b

模型 trick macro-f1 exp-no
bertweet-large 0.6714 10
bertweet-large RDrop(α=4) 0.70298 11
bertweet-large balanced focal loss + RDrop(α=4) + FGM 0.7154 14
bertweet-large balanced focal loss + RDrop(α=4) + FGM + Attention pooling + EMA 0.71128 25

subtask-c

模型 trick macro-f1 exp-no
bertweet-large balanced focal loss + RDrop(α=4) + FGM + Attention pooling 0.57562 29
bertweet-large balanced focal loss + RDrop(α=4) + FGM + 10-fold-break3 + Attention pooling 0.5701 31
bertweet-large balanced focal loss + RDrop(α=4) + FGM + Attention pooling + EMA 0.58251 32
bertweet-large balanced focal loss + RDrop(α=4) + FGM + 10-fold-break5 + Attention pooling + EMA 0.58158 37
模型 trick macro-f1 exp-no
bertweet-large balanced focal loss + RDrop(α=4) + FGM + Attention pooling + EMA 0.58251 32
roberta-large balanced focal loss + RDrop(α=4) + FGM + Attention pooling + EMA 0.5535 39
deberta-v3-large (bs8, lr2e-5) balanced focal loss + RDrop(α=4) + FGM + Attention pooling + EMA 0.4948 41
bert-large-uncased balanced focal loss + RDrop(α=4) + FGM + Attention pooling + EMA 0.488 42

第三阶段，集成不同的模型

众所周知，一般来说，机器学习模型往往有偏差。然而，当多个模型用于预测一个新项目时，预计会有较少偏差的预测。这是集合理论的基本思想，其中使用多个模型来构建一个新模型。

任务 排名 macro-f1
TASK-A 7 / 129 0.8739
TASK-B 7 / 72 0.7505
TASK-C 1 / 53 0.6238

竞赛测试阶段

任务 排名
TASK-A 5 / 84
TASK-B 8 / 70
TASK-C 8 / 63

总结

赏

谢谢你请我吃糖果

扫一扫，分享到微信

微信分享二维码
总访问量: | 总访客:

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

广州大学-研二-黄鹏龙 研究方向：[情感分析] [幽默计算]