Edge of Sanity

In training machine learning models, there is this concept of the “edge of stability”. In very hand-wavy terms, if you just barely update your ML model after it observes new data, the model won’t end up learning much. However, you also can’t push the update strength to be arbitrarily high, because that introduces higher-order effects that jeopardize the stability of your training method. In between lies a middle ground — the edge of stability — where the updates to your model are significant enough to facilitate learning, but bounded below the point of instability.

When I first learned about this topic it weirdly felt like a life lesson to me. I try to avoid living a life thats too slow, where I could be using my time more wisely. At the same time, trying to move too fast invites chaos. By taking on too many responsibilities in an attempt to maximize your velocity, you may possibly just go insane. Again there exists a middle ground. I think of this as the “edge of sanity”. Making the most of your time in terms of personal growth, learning, fulfillment, etc. all to some degree involve striking this balance.