To make it clear from the beginning, there are two ways to make a rounded rectangle with straight edges in Photoshop and I will present both of them so you may choose which one suits you best.
Method 1 – Using Rounded Rectangle Tool
1. Activate the Rounded Rectangle Tool on the Toolbar or keep Shift pressed and then hit the U key till the wanted tool is activated.
2. If you want to setup a certain roundness to the shape, change the Radius on the Properties bar then draw the rectangle.
3. Select the Convert Point Tool and click on the edge of the rectangle. You will see some small squares appear at the corners, known as anchor points.
4. Now, click on all anchor points on the edges you want to straighten. You will notice that they transform into straight lines.
5. Select the Direct Selection Tool or press A. Zoom in to see better. Press Shift while dragging with your mouse the top anchor point and stop when you see that it formed a straight edge. Do the same with the other corner. While doing these actions, make sure none of the anchor points turn black or else when dragging one node you will see the whole rectangle change.
Method 2 – Using Rectangular Marquee Tool
1. Select the Rectangular Marquee Tool or press M and create a rectangle on a new layer.
2. Go to Select – Modify – Smooth and fill in what radius you want, for instance 10.
3. The dotted rectangle has now rounded corners. With the same Rectangular Marquee Tool activated, select from the Properties bar Add to Selection.
4. Now trace at each corner a rectangle in such a way that it will form a straight edge, like in the picture below. Be careful to override the edges of the rectangle, zoom in if necessary. The fastest way to zoom in is to pres CTRL and + several times, until you are pleased with the view.
5. After you’re done with the other corner too, click on Set foreground color, pick a color and then press Alt+Backspace to fill in the Rectangle with foreground color. Press Ctrl+D to get rid of the selection.
That’s it, now you know two ways to do a rounded rectangle with straight edges.