I guess "remove or don't remove" is a personal choice on what level of detail you want to achieve. My thinking is that if the face is to be seen and is intended to be smooth I would make it smooth. As it is an inside face I may not spend as much time as on the outside but I would still smooth it out.
With respect to the squareness question. It may be a technique issue that is preventing the wood riding against the fence firmly the entire way through the cut.
With respect to the squareness question. It may be a technique issue that is preventing the wood riding against the fence firmly the entire way through the cut.