OptINC: Optical In-Network-Computing for Scalable Distributed Learning

Dimensions overall score 7.0

• The proposed Optical In-Network-Computing (OptINC) architecture can eliminate communication overhead in distributed learning compared to the ring all-reduce baseline.result 90%
  Evidencepartial

  The proposed framework can achieve comparable training accuracy to the ring all-reduce baseline, while eliminating communication overhead.

  Implicationpartial

  Explicitly stated in the abstract as a primary result of the work.

  Verificationpartial

  partial

• OptINC uses Mach-Zehnder-Interferometers (MZIs) to implement gradient averaging and quantization computations directly in the optical domain.method 95%
  Evidencepartial

  To execute gradient averaging and quantization in the optical domain, we incorporate optical devices such as Mach-Zehnder-Interferometers (MZIs) into the interconnects.

  Implicationpartial

  Directly stated in the abstract and detailed in the preliminaries and proposed work sections.

  Verificationpartial

  partial

• A preprocessing unit implemented in the optical domain reduces the dataset complexity for training the optical neural network (ONN) used in OptINC.method 85%
  Evidencepartial

  To lower the data complexity, a preprocessing unit is introduced before the ONN... reducing the ONN input size

  Implicationpartial

  Explicitly described as a proposed method to handle exponential dataset growth.

  Verificationpartial

  partial

• Hardware cost of the optical neural network is lowered by approximating its weight matrices with unitary and diagonal matrices.technical 90%
  Evidencepartial

  Hardware cost is lowered by approximating the weight matrices of the optical neural network with unitary and diagonal matrices

  Implicationpartial

  Directly stated in the abstract as a key technical contribution.

  Verificationpartial

  partial

• A hardware-aware training algorithm is proposed to maintain the accuracy of the ONN despite the hardware approximations.method 90%
  Evidencepartial

  while the accuracy is maintained by a proposed hardware-aware training algorithm.

  Implicationpartial

  Directly stated in the abstract as a key method to preserve performance.

  Verificationpartial

  partial

• Optical circuit switches (OCS) are suitable for distributed learning because communication patterns are predetermined, making their reconfiguration latency negligible.technical 85%
  Evidencepartial

  in distributed learning, the communication patterns are predetermined and require few changes during the training, making the reprogramming costs negligible.

  Implicationpartial

  Directly argued in the preliminaries section, linking a known limitation of OCS to a favorable condition in distributed learning.

  Verificationpartial

  partial

• An arbitrary M×N matrix can be implemented with MZIs by decomposing it via Singular Value Decomposition (SVD) into unitary and diagonal components.technical 95%
  Evidencepartial

  To implement an arbitrary M×N matrix W with MZIs, W is decomposed using Singular Value Decomposition (SVD)

  Implicationpartial

  Explicitly described with a formula and implementation details in the preliminaries section.

  Verificationpartial

  partial

• The proposed OptINC framework achieves comparable training accuracy to the ring all-reduce baseline on tasks like ResNet50 on CIFAR-100 and a LLaMA-based network on Wikipedia-1B.result 90%
  Evidencepartial

  The proposed solution was evaluated on real distributed learning tasks, including ResNet50 on CIFAR-100, and a LLaMA-based network on Wikipedia-1B. In both cases, the proposed framework can achieve comparable training accuracy to the ring all-reduce baseline

  Implicationpartial

  Explicitly stated in the abstract as an evaluated result on real tasks.

  Verificationpartial

  partial

Competitive landscape

OptINC offloads distributed learning computations, like gradient averaging and quantization, directly into optical interconnects to eliminate communication overhead and accelerate large model training.