US Army Special Forces: is making the cut more about physical prowess or sheer will?

I have been considering entering service in the military, and from the descriptions and information I've found, it seems like the special forces role is the one I'd be most interested in. I'm pretty good with language, and able to ess and yze situations pretty accurately, along with being able to memorize information easily, proven by my graduating high school with a 3.5 without ever really trying to hard. However, the only athletics I've ever participated in when in high school was cross-country running. Therefore I'm rather on the weak side, especially from a military perspective: 5'7" about 150 lbs. So I'm wondering which is the more important quality to bring into training: natural physical prowess or intellect and determination? It seems as though I'd be able to bulk up as long as I had the drive to get through the rigorous and numerous training courses required to make rank, but the mental ability has to be innate, right?