In the less-developed southern coast of Sao Paulo State lies the largest reserve of Atlantic Rainforest in Brazil, called PETAR (which in Portuguese stands for Parque Estadual Turïstico Alta Ribeira). As you'd expect from rainforest, the area is amazingly lush, and rich with rushing rivers, numerous waterfalls, and a hidden world of creatures teeming just out of sight.
The main draw for PETAR, though, are its nearly 300 caves. Many of the caves are closed to all but the most professional spelunkers due to liability issues, but there are a few general admission caves that need only basic caving equipment. They include Casa de Pedra, which has the largest cave opening in the world (although exploration of the cave is forbidden); Santana, which is most popular and accessible with tourists; and Caboclos, a nucleus of a few more interesting cave systems. Check the official site for camping, guide, and cost information.