This may be the crux of the issue. Why does "disadvantaged minority" change the situation in your opinion?
Let's say there were two hypothetical organizations. Both are well managed and effective private organizations intended to solve a specific problem, namely the reduction in single parent households among the poor. The #FFF Foundation serves exclusively disadvantaged whites and the #000 Group helps only disadvantaged blacks. Why is the #FFF Foundation racist and the #000 Group not?
I think it was your side of the argument that was alleging racism. It's more a question of propriety. When you include the fact that the black cohort suffers from those problems (poverty, illegitimacy) at a significantly higher rate, and when you bring in history and human nature, then the answer suggests itself.
If the #FFF group wants to help people with that specific problem, why decline to help a smaller slice of the population, where it's an even bigger problem? If it does exclude non-whites, that brings with it all the baggage of historical discrimation against non-whites.
With the #000, on the other hand, the historical baggage isn't there. It's understood that you have a minority group trying to solve a problem that's especially bad in their group. Also, it's not insignificant that #000 has the optics of black Americans helping their own, rather than blacks depending on white charity. (Even if, in reality, a lot of donations might be coming from other races.) The #FFF, by contrast, looks like a more affluent majority group (or at least a plurality group) that doesn't want to help a less-affluent minority. The optics don't make them racist, but a whites-only policy may be more trouble than it's worth.
If you ask #000 to help non-blacks, it looks as if you're asking the relatively small group of black Americans that aren't afflicted by poverty to help the relatively large number of poverty-stricken households in the United States - white, black, Hispanic, and so on.
Or, to explore this further, an Asian group which supports only Asian Americans ... racist or not? Given that Asians are a minority but one that makes whites look disadvantaged by comparison, where do they fall? Does minority status trump disadvantaged status?
That's an interesting question. Personally, I wouldn't have a problem with a minority group, even an affluent one, trying to help the less-privileged in their own ranks. They might face a lot of blowback, though, from non-Asians in their area. So it may not be the best idea in the world. Of course, a lot of minority groups do that sort of thing informally. That's not always so popular, either.
I might ask you, assuming you're OK with a charity that specifically helps Appalachians, would you be alright with one that specifically helps people from Beverly Hills, or West Palm Beach, or wherever it is that the wealthy congregate these days?
Alternately, does disadvantaged status trump majority? Whites die at double the rate of blacks from opioid overdoses and thrice the rate of Hispanics. Given this is a particular challenge which whites face at a drastically greater rate than blacks, would a whites-only opioid treatment program be acceptable? Obviously it ought not be called Klansman 4 Klean Krackers, but ...
This is like the situation with the #FFF, except now you have a far smaller group of minority people in need. A smaller percentage of smaller groups of people. Plus the baggage of historical discrimination.