Preheat the oven to 350º. Grease a 13 by 9 by 2-inch pan.
Mix together 2 cups granulated sugar, 5 eggs and milk in a bowl; add 2 teaspoons vanilla. Pour over cubed bread and let sit for 10 minutes.
In another bowl, mix and crumble together brown sugar, 1/4 cup softened butter and pecans.
Pour bread mixture into prepared pan. Sprinkle brown sugar mixture over the top and bake for 35 to 45 minutes, or until set. Remove from oven.
For the sauce:
Mix together 1 cup granulated sugar, 1/2 cup melted butter, 1 egg, and 2 teaspoons vanilla in a saucepan over medium heat. Stir together until the sugar is melted. Add the brandy, stirring well. Pour over bread pudding. Serve warm or cold.