This is entirely a matter of state law, but you raise several issues of general interest.  While most nonprofit corporation laws assume (or default to a rule) that each director shall have only one vote, many permit differential voting power if it is included in the articles of incorporation or bylaws (usually also requiring approval of the members if there are voting members).  While your pay-to-play criteria would probably not sit well with those promoting diversity, equity and inclusion, it would most likely be permissible in a state which does not have a one-person-one-vote requirement.

Another question, which is not clear to me from your description of the situation, is whether these designated “trustees” are all members of the board of directors or whether they also include others who are just donors to the organization.  If they are all directors, the board probably has the power to delegate the authority to run the building to a committee of the board, even if they go by a different name.  If they are not all directors, there is a question of what power the board has to delegate its fiduciary duty to non-directors who do not have fiduciary duty themselves.  Some states, such as Pennsylvania, allow the designation of an “other body” (again if included in the articles or bylaws) to conduct activities that would ordinarily be conducted by the board.  Most states would probably permit non-directors to serve on a committee to make recommendations, so long as the ultimate decision-making power is vested only in directors.  But it isn’t clear that all states will permit delegation of complete authority to non-directors.

You will have to find a lawyer who is knowledgeable about your state law to answer your question.  It is interesting, however, to see people trying to find creative solutions for specific unusual situations.