Planning 4 Day Ride out of Capetown... thoughts / advice appreciated

Hey There Wild Dogs - I'm popping over from Sydney in early Sept for a conference & have convinced my boss that I need to arrive 4 days early to get over the jet lag....  I've got a DL650 rented for those for days and would like to see a bit of your beautiful country.
Here's my plan.
Day 1 - CPT to Infanta along the coastal b roads (take hand powered ferry over to Wilsand to stay the night).
Day 2 - Prince Albert via Mossel Bay, George, Oudtshoorn (I hear the pass from Oudtshoorn to Prince Albert is legendary)  (Is this too big a day?)
Day 3 - To Barrydale via Laingsburg - too the the high country.
Day 4 - back to CPT via Robertson, Worcester.

I'm keen on a relaxed trip so I can look around with a bit of dirt to qualify as an adventure ride.

Questions:  Am I being too ambitious?  Last trip was Argentina which I totally underestimated.  Ended up doing multiple 9 hour days just to pack it in.  Yuck.

My SA mates say I simply must see Knysna and the pass behind it.  Would this add too much?

Are there any other towns I should go to?  Is Infanta a bit too small?

What about personal security?  Any tips?

So many questions.  Your thoughts much appreciated.

Drew.