I don't think history is that complicated though defending it may be. In his memo, (which I'd previously read), Rehnquists mentions previous cases getting the court into trouble without stating a view. Apparently he had a problem with SCOTUS interfering with Dred Scott in which the court said Black people had no rights the court needed to honor.

His reason for wanting to uphold Plessey had nothing to do with not wanting to get involved with individual or minority rights. He seemed to still not recognize the rights of Black people. That he didn't think segregation was extreme enough to merit the court's involvement can only come from a place of racism as have many of the courts previous and recent decisions.