I refer to Sri Lanka by its colonial name of Ceylon with special purpose. As a tourist I found that the more rewarding places have very little to do with either modern Sri Lanka or the Buddhist Sinhalese culture upon which it is founded. While Columbo and the cultural center of Kandy certainly have sights for people with specific interests, an international traveler is likely to find that they do not compare favorably with other possible destinations.
Aside from these areas, Sri Lanka has much to see. The Indian Ocean beaches along the south coast are beautiful, relaxing, and quite affordable. The old Dutch city of Galle, with its fantastic ramparts and breathtaking views, is definitely worth a visit. The Rahunu-Yala National Park offers some of the most impressive wildlife in Asia -- of a comparable quality to East African destinations.
The British left behind an antiquated, but charming railway network that serves the highlands and the southern coast. From the train, you can get some of the best views of the island, especially on the mountainous route from Colombo to Kandy. It is worthwhile riding the observation car to the highlands, even if you aren't interested in seeing Kandy.
The touts in Sri Lanka are merciless. They will relentlessly follow you, denying that they are interested in your money, and claiming that they want to be your friend. This is inevitably followed by an insistance that you come visit a cousin's shop, or go on a friend's guided tour. Be prepared for this behavior, and learn quickly to be polite but firm in your desire to be left alone.
Likewise, every transaction outside the larger hotels is subject to ruthless bargaining. After agreeing on a price, be sure to reapeat it so there is no misunderstanding. Since English is widely spoken in Sri Lanka, you are likely to encounter difficulty only when a salesman with whom you are negotiating does not wish to understand.