You can now choose to have Circleback only join meetings you create on your calendar. Head over to Settings → Calendar to find the new toggle.

With it enabled, Circleback will, by default, only automatically join meetings created by you and not those you were invited to. You can always override the automatic joining behaviour for individual upcoming meetings from Home.