I made a quick video for how to get the Griffin School Witcher set without even having to fight anything. The Green Witcher sets are the best gear in the game when fully upgraded, so I would highly recommend for you to find all the diagrams!
Start out by either buying a map from the armorer in the village of Midcopse in western Velen (this will start the Scavenger Hunt: Witcher Griffin School quest) or by heading directly to the village of Downwarren, and then exploring the question mark to the west.
The question mark will reveal a cave, inside of which you can find the 4 Griffin armor diagrams – you can either fight into the cave to get this, or reveal the secret back entrance that is being hidden by an illusion (check in the video).
After this, read the notes you looted along with the diagrams, and you will get 2 new locations highlighted on your map, where you can pick up the steel and silver Griffin sword diagrams.