Line an 8x8 baking dish with parchment paper. In a medium saucepan, over low heat, melt the butter. Add the brown sugar and evaporated milk. Stir constantly until the mixture begins to boil and reaches 240 degrees F on a candy thermometer. Remove from the heat and pour into a mixing bowl. Add the vanilla extract and powdered sugar. Mix with an electric mixer on low until the sugar is incorporated, about 3-5 minutes. Pour fudge mixture into the prepared dish. Refrigerate for at least 30 minutes. Remove from fridge, then remove from baking dish by pulling up on the parchment paper. Cut into one-inch squares.