VLSP 2022 ASR Challenge: Task-1

Organized by vlsp_2022_asr_organizer - Current server time: Oct. 7, 2022, 3:25 p.m. UTC

Current

Public Test
Sept. 19, 2022, midnight UTC

Next

Private Test
Oct. 31, 2022, midnight UTC

End

Competition Ends
Oct. 31, 2022, 8 a.m. UTC

Our website

https://vlsp.org.vn/vlsp2022/eval/asr

Important dates

  • July 27, 2022: Registration opens

  • Aug 31, 2022: Registration closes

  • Sept 05, 2022: Dataset building starts

  • Sept 12, 2022: Dataset building ends

  • Sept 19, 2022: Training data and public test data for the ASR-T1 released

  • Oct 31, 2022: Private test set released for both ASR-T1 and ASR-T2 

  • Oct 31, 2022: Test result submission

  • Nov 15, 2022: Technical report submission

  • Nov 26, 2022: Result announcement (workshop days)

General Description

Task-01 (ASR-T1): Focusing on a full pipeline development of the ASR model from scratch with limited training data condition. The organizer will provide two training datasets. The first dataset is around 5o hours of transcribed data. The second dataset is around 400 hours of untranscribed in-domain data. All participants are required to use only this provided data to develop models including acoustic and language models. Any use of another resource for model development is not acceptable. 

Task-02 (ASR-T2): Focusing on spontaneous speech in different real scenarios e.g., meeting conversation, lecture speech. For this task, the organization will not provide training data, participants can use all available data sources to develop their models without any limitation.

Evaluation metric

The quality of the models will be evaluated by the Syllable Error Rate (SyER) metric.

SyER = (S+D+I)/N

where

  • S is the number of substitutions,
  • D is the number of deletions,
  • I is the number of insertions,
  • C is the number of correct syllables,
  • N is the number of syllables in the reference (N=S+D+C)

 

 

- no title specified

General rules

  • Right to cancel, modify, or disqualify. The Competition Organizer reserves the right at its sole discretion to terminate, modify, or suspend the competition.

  • By submitting results to this competition, you consent to the public release of your scores at the Competition workshop and in the associated proceedings, at the task organizers' discretion. Scores may include but are not limited to, automatic and manual quantitative judgments, qualitative judgments, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

  • By joining the competition, you accepted to the terms and conditions of Agreement form of VLSP 2022 - ASR, which has been sent to your email. It is noted that your participant rights will be revoked if you do not sign and send back to us before the deadline.

  • By joining the competition, you affirm and acknowledge that you agree to comply with applicable laws and regulations, and you may not infringe upon any copyrights, intellectual property, or patent of another party for the software you develop in the course of the competition, and will not breach of any applicable laws and regulations related to export control and data privacy and protection.

  • Prizes are subject to the Competition Organizer’s review and verification of the entrant’s eligibility and compliance with these rules as well as the compliance of the winning submissions with the submission requirements.

Eligibility

  • Each participant must create a CodaLab account to submit their solution for the competition. Only one account per user is allowed.

  • The competition is public, but the Competition Organizer may elect to disallow participation according to its own considerations.

  • The Competition Organizer reserves the right to disqualify any entrant from the competition if, in the Competition Organizer’s sole discretion, it reasonably believes that the entrant has attempted to undermine the legitimate operation of the competition through cheating, deception, or other unfair playing practices.

Team

  • Participants are allowed to form teams. The maximum of the number of participants on the team is up to 5.

  • You may not participate in more than one team. Each team member must be a single individual operating a separate CodaLab account.

Submission

  • Maximum number of submissions in each phase:

    • Phase 1 - Public Test: 1 submissions / day / team

    • Phase 2 – Task 1 - Private Test: total 2 submissions / team.

    • Phase 3 – Task 2 - Private Test: total 2 submissions / team.

  • Submissions are void if they are in whole or part illegible, incomplete, damaged, altered, counterfeit, obtained through fraudulent means, or late. The Competition Organizer reserves the right, in its sole discretion, to disqualify any entrant who makes a submission that does not adhere to all requirements.

Data

By downloading or by accessing the data provided by the Competition Organizer in any manner you agree to the following terms:

  • You will not distribute the data except for the purpose of non-commercial and academic-research.

  • You will not distribute, copy, reproduce, disclose, assign, sublicense, embed, host, transfer, sell, trade, or resell any portion of the data provided by the Competition Organizer to any third party for any purpose. 

  • The data must not be used for providing surveillance, analyses or research that isolates a group of individuals or any single individual for any unlawful or discriminatory purpose.

  • You accept full responsibility for your use of the data and shall defend and indemnify the Competition Organizer, against any and all claims arising from your use of the data.

Submission Guidelines

Submission Format

The result to upload to AI Hub is a zip file containing single file "results.tsv" that conforms bellow the specifications.

Submissions have to be made in UTF-8, lower-case and one line for each utterance.

utterance_name<TAB>recognized_text_sequence

For example

0001.wav chào mừng các bạn đã tham dư cuộc thi

0002.wav tên tôi là nguyễn văn a

Output Conventions

Since there are cases that input speech can be interpreted in different ways, the below rules are applied to mitigate such an issue:

  1. Numbers, dates etc. need to be transcribed in words as they are spoken, not in digits

  2. Common acronyms such as nato, fifa, are written as one word, without any special markers between the letters. This applies no matter whether they are spoken as one word or spelled out as a letter sequence. All other letter spelling sequences are written as individual letters with space in between.

  3. For English words, names of people and places in other languages such as youtube, facebook, are written as it, not in Vietnamese pronunciation.

Public Test

Start: Sept. 19, 2022, 7 a.m.

Private Test

Start: Oct. 31, 2022, 7 a.m.

Competition Ends

Oct. 31, 2022, 3 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 vlsp_2022_asr_Lightning 13.37
2 duongta007 22.81
3 vlsp_2022_asr_STanDard 24.28