Preheat oven to 350 degrees F.
Grease and lightly flour an 8 ½ x 4 ½ inch loaf pan.
In a large bowl of an electric mixer beat together butter and Swerve granular.
Add the whole egg and egg whites one at a time, beating well after each addition.
Beat in lemon zest, lemon juice, vanilla, and yogurt.
In a separate bowl sift together the flours, baking powder, and salt.
Add ½ the flour and ½ the milk to the butter mixer and stir on low speed.
Add in the remaining flour and milk and continue to stir on low speed until combined.
Pour batter into prepared loaf pan and tap pan to level the batter.
Bake lemon loaf for about 1 hour, or until a cake tester inserted into the center of the loaf comes out clean.
Cover the lemon loaf with aluminum foil ½ way through the baking process.
Cool loaf for 30 minutes.
Remove lemon loaf from the loaf pan. You may need to run a knife along the edge of the loaf to safely remove it from the pan.
Prepare the glaze. You can glaze the lemon loaf while it is still warm or after the loaf has cooled.
Allow the glaze to harden before slicing.