Good question! So, you can use T-connectors or 90s which is what you have by the sound of it. I like Ts because they are so basic and versatile, but the 90 is a better choice for nice, rigid corners.
You are right, the 90s need that third perpendicular pipe to make everything nice and tight. Just take a scrap of conduit and cut a short stub as long as the width of the connector and put that in the corner. For a lot of builds, you need that third pipe to complete a vertex like the corner of a cube which you cannot do with two Ts without a bit of offset. In your case you will end up with nice corners, but yeah, you will need that short piece to fill in the gap.