It’s alarming that the OP calls the child James’. It’s not James’ kid. It’s never been HIS child and never will be. That is not “the only grandchild” that family has no grandchildren because the child belongs to his two parents, Miranda and Vivian. There is a shit ton of assumptions here by the OP that is very alarming. They suck for weaponizing access to the child, providing a reason to not want to be further involved. They’re disrupting the existing child’s family structure over a disagreement – that needs to be sorted out. I hope there is a relationship counselor involved to help the non-traditional family navigate things. Their actions with the child make me think there isn’t.
This one is really hard, IMO. I get where you’re coming from, and I don’t think your feelings are invalid by any means. I’m imagining a worst-case scenario where God forbid, you find you can’t get pregnant, and then you have two women actively present in your life who have both had no trouble getting pregnant with your husband’s sperm. If I was in that position, I wouldn’t be able to take it. I’d grow resentful of my husband and of the two women, I’d feel like my body failed me. That’s a massive what-if, but it would be hard enough to handle with the one son already existing, let alone another newer one that’s born around the time you thought your own child would have been born.