The politician lost the Faizabad constituency in which he built the controversial Ram temple