Agreed with the colour of skin shouldn't have any effect on how good a referee you are but also why does it matter if every referee is white or black or Asian or whatever. It should be on efficiency of the job at hand.
Personally (and sometimes I need to get my tin hat on for some reason) I disagree with female officials refereeing men's games and vice versa. The reason for me is that with female officials what ultimately you are saying is that the pinnacle of being a female referee is to referee in the men's game whether that be football league or Premiership. You are saying the women's game is not good enough for you as a referee you would be better in the men's game. it has the potential to restrict development of the women's game.
If you look at players it is estimated the average pay in the Women's Super League is £30,000 per year vs £2,800,000 for Men in the EPL
If you look at the referee salary is it estimated to be around £70,000 (including a "retainer" of circa £40,000) in the EPL (currently all men) something on the figures need to change as if you were bringing in regular women referees into the men's game they would be being paid more than double than the average player in the WSL. Rather than handpicking a 1 or 2 out and sticking the odd one into the men's game the FA in collaboration with other partners needs to develop the women's game with extra funding, pay the players more and use the good female referees to develop the female game.
When I have had a similar chat to the above before with friends, I was a referee, I was at both Leeds and Sheff United (twice) Academies refereeing U12-U16s age groups along with open age Saturday and Sunday football. You don't really do it for the money but it certainly helped when I was 17/18 - £15-£20+ mileage could easily get a ticket to BDTBL or most away games too but when you are on you have a game on your doorstep and you are happy to go to the local park on a Sunday do your bit, yeah you might get some stick from some bloke who is near on 40 and 20 stone and hoofs the ball but has a go at you etc, typical Sunday morning stuff but you could have some banter and usually a bit of a laugh. I then get told no you can't do that game (this actually happened to me) by the local county FA I was expected to travel a circa 80 mile around trip to run the line on a 10am kick off on a U12's girls game (30 mins each way). Meaning I would have to be there for 9am on a Sunday setting off at 8am not getting home till 12:30pm if lucky with Sunday morning traffic on the way back. For 60 minutes of girls football for just mileage at £0.25 per mile (£20) (no actual pay) when I said I wouldn't do it, I got suspended from refereeing as I did the fixture I wanted to do which was a 5 min drive from my house. Never done a game since.