Preheat the oven to 400°F. Butter the bottom and sides of a 9-inch baking pan. Arrange the peaches in the pan. Set aside.
Combine the sugar, sea salt, flour, and rosemary in a medium bowl. Add the vanilla, buttermilk, and eggs, and whisk until combined. Stir in the butter.
Pour the batter over the peaches and bake until cooked through, approximately 40 minutes. Remove from the oven and cool on a rack for 10 minutes. Invert the cake pan onto a large plate. Sprinkle the confectioners’ sugar over the top and garnish with additional rosemary if desired. Serve immediately.