In reply to Kaizen:
You can go a little further with this exercise to strengthen the back. If you know what 4 point kneeling is, you can do this with the ball on your stomach lifting opposite arm and leg. You can also progress to lifting both legs up with arms in pushup position. Make sure you clench your glutes before lifting each leg. This is something my physio has me doing for my back problems.
Sitting on the ball and doing bicep curls and overhead tricep extensions with dumbbells are good. Lie back on the ball on your shoulders and do presses (like bench presses) with dumbbells.
Make sure you have balance in your routine. If you are doing press ups, make sure you are doing some kind of rowing (pulling) exercise along the same plane of motion on a 1:1 ratio. If you are doing chinups do overhead presses. If you are doing hangs do handstands.
Maybe look into kettlebells - swings and circles are 2 good exercises for healthy shoulders.
Hope this helps
Seb