For convex polygon K, with vertices A1, A2, A3,....,A20.
Total number of ways to select three sides in 20 sides of k polygon ={20 \choose 3}
=\frac{20!}{17!3!}=\frac{20\times19\times18}{6}=1140
now, number of ways of selection when exactly three sides, are common ({A1A2,A2A3,A3A4} type all selection)=20
now, number of ways of selection when exactly two sides are common{A1A2,A2A3} then we can't select A3A4 and A1A20}=20 \times 16=320
again number of ways of selection when the selected two sides are A1A2 and A3A4(one side gap between them)=20
Two cases possible for third side
case1 third side selected at a gap of two from above selected two sides
{if we take A1A2,A3A4 then we can not choose 3rd side A3A4, A4A5,A20A1, A19A20}
number of ways of third side section=20-6-1=13
required ways=20 \times 13=260
case2 3rd sides is at a gap of one from any selected side=20
so number of ways=1140-20-320-260-20=520