2024 10 03 first step to autonomy

This week, we took a major leap toward full autonomy in deep learning workflows!

Monday, September 30, 2024: Tackling Overfitting with Spark

On Monday, we recorded a demo showcasing how Spark, our Large Language Model (LLM), assists in addressing overfitting during an experiment (a supervised learning training session). The user kicked off the experiment earlier and is now asking Spark to analyze the run. After the user agrees with Spark's analysis, the user instructed Spark to make the necessary changes to the experiment to mitigate overfitting.

Watch as Spark automatically adjusts the hyperparameters, stops the experiment, makes the changes, and then resumes training—all while the user monitors progress:

Here's a high-resolution screenshot capturing the final screen of the reactive demo:

Wednesday, October 2, 2024: Spark Goes Fully Autonomous

By midweek, we captured another exciting demo where Spark took even more initiative. After the user started a new experiment, Spark, without prompting, analyzed the experiment after only two epochs. Noticing signs of underfitting, Spark automatically informed the user of the issue and then took corrective action. It paused the experiment, made the necessary adjustments, and resumed training—all autonomously.

The results? A significant improvement in validation loss, visible in the very next epoch. Witness the magic here:

Here's a high-resolution screenshot capturing the final screen of the proactive demo: