In November 2022, GTCOM released the "GeWu" big model which updated the KgCLUE1.0: Large-scale Knowledge Graph Question Answering List and Named Entity Task List in CLUE, China’s most authoritative benchmark in the field of NLU, and won the first place in the leaderboard of both lists. "Gewu" in Chinese means to investigate things/affairs, and represents GTCOM’s persistent effort in the artificial intelligence field.

CLUE (www.cluebenchmarks.com) is recognized as the most authoritative evaluation benchmark in the field of natural language understanding in China. It attracts many industry-leading enterprises such as Tencent, Huawei, and Alibaba and research institutes to participate in the evaluation. The leaderboard is highly competitive and is a must-win for natural language understanding teams in the industry.

GTCOM won the first place in the KgCLUE1.0 large-scale knowledge graph question answering task list.

GTCOM won the first place in the CLUE named entity task list.

GeWu is a multilingual and multimodal foundation model technology developed by GTCOM since 2021, serving as a base for multilingual intelligent processing and application. It includes the pretrained multilingual model, pretrained multimodal model and large-scale model for multilingual machine translation. The GeWu pretrained multilingual model, which topped the CLUE leaderboards, represents GTCOM’s industry-leading position in the field of foundation model technology.

The GeWu pretrained multilingual model takes advantage of GTCOM’s large-scale multilingual language resources and uses a multilingual pre-training method based on knowledge control information embedding to realize multi-round two-way driving and fusion of large-scale multilingual non-aligned unlabeled data, bilingual aligned sentence pairs and cross-language knowledge data. It supports more than a hundred languages. The model scale covers a variety of parameters such as lite, medium and large scales (with tens of billions of parameters), while innovatively combining multiple languages, tasks, and scenarios into a pluggable and flexibly scalable unified downstream task framework to support multi-scenario learning tasks such as natural language understanding, natural language generation, and knowledge graph.

Meanwhile, the GeWu Large-scale Model for Multilingual Machine Translation was also released. It adopts a large-scale pre-trained method based on a new mixture of experts (MoE), innovatively proposes a MoE with ring-shaped attention, breaks the constraints of model capacity and multiple languages, and realizes multilingual machine translation under a single model. It reaches the ultra-large scale of hundreds of billions of parameters, and fundamentally improves the basic performance of low-resource multilingual machine translation. The project was shortlisted for the 2021 Key Tasks for Innovation in New-Generation Artificial Intelligence by the Ministry of Industry and Information Technology, and the technological achievements were selected into the 2022 Best Practices of the Application of Large-scale Pre-trained Modelby the Artificial Intelligence Industry Alliance.

GTCOM is currently working on the GeWu pretrained multimodal model. The model uses massive text-image paired data, and Vision Transformer and CLIP (Contrastive Language-Image Pre-training) to realize unified modeling of images and text and achieve multilingual cross-modal semantic alignment. It adopts the deep diffusion network and produces a graphic-text generation model with more than 10 billion parameters. The model is expected to be officially released in the first quarter of 2023.

GTCOM is a leading big data and AI company. With independently developed advanced technologies such as machine translation, scientific research data analysis, financial technology and digital city brain, it provides a full range of leading big data and AI scenario-based solutions for governments and enterprises worldwide.