Preheat the oven to 350°. Generously butter a 13 by 9-inch baking dish.
In a medium saucepan, combine the milk, sugar and 1/2 cup brown sugar divided, sweet potato puree, vanilla, cinnamon and nutmeg. Cook over medium-high heat, just until the mixture begins to boil. Remove from the heat and set aside.
In a medium bowl, whisk the egg yolks until smooth and combined. Using a ladle, pour 1/2 cup of the hot milk mixture in a slow, steady stream into the yolks, whisking constantly. Add another 1/2 cup of the hot milk mixture and whisk to combine. Pour the egg mixture into the milk mixture. Cook over medium-high heat, stirring constantly, until it's thick enough to coat the back of a wooden spoon, about 3 to 5 minutes. Strain the custard and reserve.
Cover the bottom of the baking dish with 1 layer of bread slices. Pour about 1/4 of the custard mixture over the bread. Repeat with remaining bread and custard, ending with custard. Cover and refrigerate for 8 hours or overnight.
Sprinkle the bread pudding with the pecan crumble. Bake until the crumble mixture is golden and the custard is bubbly, about 45 minutes. Serve warm.
In a medium bowl, combine flour, 1 cup brown sugar, butter, oats and pecans and mix with a hand mixer on low speed, until well combined.