Season the catfish fillets on both sides with salt and pepper. Mix together buttermilk and hot sauce in a 8x8 casserole dish. Add the catfish fillet, making sure they're completely covered by the liquid. Let sit for 15 minutes.
Meanwhile, heat oven to 475°F.
Coat a sheet tray with the olive oil. Add to the oven and heat while you prep the remaining ingredients.
Whisk together the flour, cornmeal, and crab boil in another casserole dish. Remove the catfish from the buttermilk, letting excess drip off, and dredge on both sides with the cornmeal mixture. Transfer to the hot baking dish. Repeat with the 3 remaining fillets. Bake for 6 minutes, then carefully flip the catfish, and continue cooking for 5 minutes more, or until cooked through and golden.