Do you think this is a bad idea?

I think your idea is the most sensible. Maybe if this was your first and you didn't have to worry about other kids. I think this would work best for the kids as well so they don't feel left out or ignored.