1. Background

2. Problems

2.1. Problem 1: Single Level of Abstraction

• Human Intelligence: All current LLMs miss a crucial characteristic of human intelligence: explicit reasoning and planning at multiple level of abstractions, not only the word level.
  • Although the low-level expressions might vary, the flow of higher-level abstract ideas will remain the same.
  • Humans also detect and remember dependencies between the different parts of a longer document at an abstract level.
  • Humans use a hierarchical approach to processing and analyzing information.
• LLM Intelligence: Current LLMs can’t support the explicit hierarchical structure of information processing and generation, with any particular language or modality.

3. Hypothesis

3.1. Hypothesis 1 (on Problem 1): Sentence-Centric Modeling

• A sentence (concept) is an appropriate unit to achieve language independence, in opposition to single words.

4. Solutions

4.1. Solution 1 (for Problem 1): Concept Modeling

4.1.1. Two Levels of Abstraction

• Subword tokens
• Concepts
  • A sentence in a text document
  • An equivalent speech utterance

4.1.2. Data Preparation

• Challenges

  • Accurate segmentation of sentences is challenging because …
    • There are errors, specific formatting issues, other source of noise.
    • Some sentences are very long and complex.

• Sentence Segmentation Analysis

  • SpaCy Segmenter Capped
    • Rule-based segmenter.
    • Break down long sentences into shorter segments based on punctuation marks.
  • SaT (Segment any Text) Capped
    • Model-based segmenter.
    • Leverage the provided splitting probability estimates to identify the next best split.

• Efficacy Metric: AutoBLEU

  • Compare the segment decoded from the SONAR decoder after encoding the original segment, against the original segment.
  • A good segmentation call yield segments that can be encoded and decoded without loss of signal.
  • The paper selects SaT Capped with the capped sentence length of 200 characters.

  

4.1.3. SONAR Sentence Embedder

• SONAR Training Architecture

  • Encoder/Decoder Bottleneck Architecture: Train on text inputs.
    • Training criterion: $\mathcal{L}{\text{MT}} + \alpha \cdot \mathcal{L}{\text{MSE}} + \beta \cdot \mathcal{L}_{\text{AE/DAE}}$
      • $\mathcal{L}_{\text{MT}}$ is the machine translation objective of 200 languages in and out of English.
      • $\mathcal{L}_{\text{MSE}}$ is an explicit MSE loss at the embedding bottleneck layer.
      • $\mathcal{L}_{\text{AE/DAE}}$ is a denoising auto-encoding task.
  • Teacher-Student Approach: Extend to speech inputs.
    • Training criterion: $\mathcal{L}_{\text{MSE}}$ between teacher embeddings and student embeddings.