Will there be more optimization example tutorials? #5
Description
I am very grateful to the author for publishing this tutorial; I have learned a lot from it.
If possible, I hope the author can add more training optimization tutorials in the future. This would provide readers with more tuning ideas and help them understand the concepts. For example, how to optimize distributed training scenarios and how to optimize computation and communication overlap.
When I was training a VLA model using DeepSpeed zero2, I found that computation and communication did not overlap well, as shown in the figure below. As a beginner, I am not sure how to optimize it (besides modifying the DeepSpeed configuration).
