Did Musashi wait 3 years to fight the Yosjioka

On a reread and just finished the Tekezou arc. In the beginning panels if the next arc, it says that he's in his 21st year. Did Musashi wait 4 years to battle someone new after the hanging in the tree?