Ousted chairman and founder of Men’s Wearhouse quits board