Analysts believe Kayani was reluctant to cross Afghan militants with whom Pakistan has historical ties and could be valuable allies in Afghanistan after foreign forces withdraw.
The Afghan and Pakistani Taliban are allies, but have largely focused their attacks on opposite sides of the border.
Pakistan has experienced three military coups and has been run by the army for half of its 66-year history. Kayani's stance allowed the country to experience its first successful transition between democratically elected governments earlier this year although the army has continued to play a powerful role in politics in the background, especially with regard to policy toward the Afghan war and Pakistan's archenemy India.
"I think the army will continue to stay out of politics, and I think the power of the army will continue to get diluted over time," said Talat Masood, a retired army general and defense analyst.
The prime minister's office announced the appointment in a statement sent to reporters. Gen. Sharif most recently served as the head of the army's training and evaluation wing, an important position in a country where the force has attempted to retool its skills toward counterinsurgency. Several of his family members have served in the army. His brother, who was killed in the 1971 war with India, was one of the force's most decorated soldiers.
The new army chief also served as head of an infantry division in Lahore, the capital of Punjab province, home to the prime minister. That may have been one of the factors that influenced the prime minister's decision to choose him, said Masood, the defense analyst.
"I think there were several factors he had to bear in mind when selecting the chief," said Masood. "One was that he will distance himself from politics and not be a potential threat to him. The most important factor was how professional he was, how acceptable he was in the army."
U.S. State Department spokeswoman Jen Psaki said the U.S. looks forward to working together with the general.
Gen. Sharif is taking over over at a difficult time for the army after years of battling the Pakistani Taliban. The government recently tried to jumpstart peace talks with the Pakistani Taliban, but the effort was torpedoed earlier this month by a U.S. drone strike that killed the group's leader.
Pakistan also has been a strong proponent of peace talks between the Afghan government and the Afghan Taliban and has released around four dozen militant prisoners in an attempt to help negotiations. But the releases don't seem to have done much good so far.
Sharif was third in line for the post of army chief in terms of seniority. The prime minister passed over the candidate considered to be Kayani's favorite, Lt. Gen. Rashad Mahmood, who was appointed chairman of the joint chiefs of staff, considered to be a less influential job.
The prime minister also passed over the most senior candidates when he chose Musharraf as army chief in 1998. He was toppled in a coup by Musharraf in 1999 after he tried to fire him.
The premier spent years in exile in Saudi Arabia before returning in 2007. His party won national elections earlier this year, and he took office in June.
Musharraf is currently facing treason charges filed by the government a major challenge to the army's once untouchable position.
Also Wednesday, President Mamnoon Hussain selected Tassaduq Jillani to be the next chief justice of Pakistan's Supreme Court, said two officials in the president's office, speaking on condition of anonymity because a formal statement had not yet been issued. Jillani is the second most senior judge on the court after the current chief justice, Iftikhar Chaudhry, who is scheduled to retire in December.
Chaudhry has faced both praise and criticism as perhaps the most activist chief justice in the country's history.