I have worked with 4 shots that do such pans. Initially I thought it was the motion blur from my phone that caused it, but I've also returned to the locations to film them in 4k60fps and in slow motion, and I even moved slowly. 118 user tracks later, and C4D still loses track somewhere randomly at 25%, 50% or 75% of the pan. I've also checked each and every user track and deleted bad auto tracks.
Please have a look at my footage: https://drive.google.com/file/d/1DQhWpVvqVWV8t-i_w_ejbQL9wLolNp6Z/view?usp=sharing (please to convert the footage since attaching the PNG sequence would make the