how to key out a particular area in a video

I am a frame from my video here



I want to key out only the whiteboard part and slot an image layer underneath.

I am wearing a white shirt, which got affected when I apply the keyer.

I don't want to put the image layer above as there will be times my hands goes over the whiteboard. 

Do you all know if there is a way to key out only the whiteboard part? Thank you. 

Some cheap green cloth, double-sided tape, Bob's your uncle.