It could potentially violate conflict rules. (The idea being that the lawyer might have been privy to your secret information while his old firm represented you.)