Most paintings that we see have traditionally been square or rectangular in shape. I don’t think there is any definitive reason why this is so except that rectangles are an easy shape to work on and remind us of looking through a window in a rectangular wall. But perhaps it is also something to do with the fact that it is relatively easy to stretch a canvas in the two directions needed to cover a four sided frame. Stretching it in more directions can cause slack areas on the canvas making it uneven and unsuitable for painting on.
In addition round, triangular and other polygon frames are expensive, tricky to make and difficult to keep stable. It is possible to frame a round canvas in a square frame but then it adds four additional triangular shapes to each corner of the image.
It has been done though and some well known painters have created famous works of art, often images of the Madonna, on round canvases. This painting is by Michelangelo from around 1507.
The lenses we use in photography are round and it makes sense that the image formed by these lenses is also round yet we still generally display photos square or as 2×3 rectangle.
The camera obscura was the first way of creating photographic type images. Originally used thousands of years ago by astrologers to study the sun without hurting their eyes. By darkening a room so that it is completely dark except for a single tiny hole in one wall, light is bounced off the objects outside and projected on the wall opposite. It is upside down and back to front but is a perfect replica of the scene outside.
In around 1550 an Italian called Girolamo Cardano inserted a lens into the camera obscura’s pinhole. The lens needed focusing but it produced a much sharper and brighter image than had been possible before. Later, camera obscuras became seaside attractions and several are still open for the public to visit.
From the image above you can see that the reflected image is perfectly round. This gives a really good example of how images are formed in a camera. Light is reflected off whatever is in front of the camera, back through the lens(es) and onto the film/sensor inside.The lens bends the light so that it focuses perfectly on the film/sensor. Because light needs to bend further for it to reach the edges of the image it causes more distortion (aberrations) there than it does in the centre of the image. To get the best quality therefore the camera crops out the softer focused areas at the edges of the images.
There are also more practical reasons why I think photographs are rectangular. I think that square plates and slides were initially easier to produce, handle and store for early photographers and then when film was made on a roll, it made much more sense for the images it recorded to be rectangular rather than circular. I also think that simply having seen square and rectangular paintings that square and rectangular photographs were what people expected to see.