The short answer is it’s certainly possible, but I don’t know if that’s what happened in your case. Ghana is close enough geography-wise that I imagine there were interactions between darker and lighter-skinned Africans for centuries, and differences could have been tribal or otherwise. I also am not familiar with the colonization of Ghana and their eventual independence, nor current politics. I could speculate off what I think I know, but it would only be guessing.