Moderating Illicit Online Image Promotion for Unsafe User-Generated Content Games Using Large Vision-Language Models

📄 arXiv: 2403.18957v2 📥 PDF

作者: Keyan Guo, Ayush Utkarsh, Wenbo Ding, Isabelle Ondracek, Ziming Zhao, Guo Freeman, Nishant Vishwamitra, Hongxin Hu

分类: cs.CY, cs.CL, cs.LG, cs.SI

发布日期: 2024-03-27 (更新: 2024-08-12)

备注: To Appear in the 33rd USENIX Security Symposium, August 14-16, 2024

💡 一句话要点

提出UGCG-Guard以解决不安全用户生成内容游戏的非法推广问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 用户生成内容 图像识别 视觉-语言模型 在线安全 社交媒体监管 儿童保护 机器学习

📋 核心要点

  1. 现有研究对UGCGs非法推广的关注不足，缺乏全面的训练数据和针对性的解决方案。
  2. 本文提出UGCG-Guard系统，结合大型视觉-语言模型和条件提示策略，旨在自动标记非法UGCG推广内容。
  3. UGCG-Guard在真实场景中检测准确率达到94%，显著提升了对不安全内容的识别能力。

📝 摘要（中文）

在线用户生成内容游戏（UGCGs）在儿童和青少年中日益流行，但也带来了对显性内容的暴露风险，尤其是在社交媒体上。尽管存在这些担忧，针对UGCGs非法推广的研究仍然较少。本文首次收集了2924张展示性别和暴力内容的图像，揭示了这一问题的紧迫性，并提出了UGCG-Guard系统，利用大型视觉-语言模型（VLMs）和新颖的条件提示策略，实现了94%的检测准确率。

🔬 方法详解

问题定义：本文旨在解决在线用户生成内容游戏（UGCGs）中非法推广的图像识别问题。现有方法难以获取全面的训练数据，且UGCG图像与传统不安全内容存在显著差异。

核心思路：论文提出的UGCG-Guard系统利用大型视觉-语言模型（VLMs）进行图像识别，并采用条件提示策略实现零-shot领域适应，结合链式思维推理进行上下文识别。

技术框架：UGCG-Guard系统主要包括数据收集模块、模型训练模块和图像识别模块。通过收集2924张图像，构建训练集，并利用VLMs进行特征提取和分类。

关键创新：UGCG-Guard的创新在于结合了条件提示策略和链式思维推理，这使得系统能够在缺乏大量标注数据的情况下，依然实现高效的图像识别。

关键设计：系统在参数设置上进行了优化，采用了适应性损失函数以提高模型的泛化能力，并设计了特定的网络结构以增强对UGCG图像的识别效果。

🖼️ 关键图片

img_0

📊 实验亮点

UGCG-Guard在真实场景中的检测准确率达到94%，显著高于现有方法，展示了其在识别非法UGCG推广图像方面的优越性能。这一成果为社交媒体平台提供了有效的工具，以应对日益严重的在线安全问题。

🎯 应用场景

该研究的潜在应用领域包括社交媒体平台、在线游戏监管和儿童保护技术。UGCG-Guard能够有效识别和标记不安全内容，帮助平台维护用户安全，减少儿童和青少年接触不当内容的风险，具有重要的社会价值和实际意义。

📄 摘要（原文）

Online user generated content games (UGCGs) are increasingly popular among children and adolescents for social interaction and more creative online entertainment. However, they pose a heightened risk of exposure to explicit content, raising growing concerns for the online safety of children and adolescents. Despite these concerns, few studies have addressed the issue of illicit image-based promotions of unsafe UGCGs on social media, which can inadvertently attract young users. This challenge arises from the difficulty of obtaining comprehensive training data for UGCG images and the unique nature of these images, which differ from traditional unsafe content. In this work, we take the first step towards studying the threat of illicit promotions of unsafe UGCGs. We collect a real-world dataset comprising 2,924 images that display diverse sexually explicit and violent content used to promote UGCGs by their game creators. Our in-depth studies reveal a new understanding of this problem and the urgent need for automatically flagging illicit UGCG promotions. We additionally create a cutting-edge system, UGCG-Guard, designed to aid social media platforms in effectively identifying images used for illicit UGCG promotions. This system leverages recently introduced large vision-language models (VLMs) and employs a novel conditional prompting strategy for zero-shot domain adaptation, along with chain-of-thought (CoT) reasoning for contextual identification. UGCG-Guard achieves outstanding results, with an accuracy rate of 94% in detecting these images used for the illicit promotion of such games in real-world scenarios.