Let's go back a bit further.
What does the Nicene Creed mean when it affirms that the church is "one, holy, catholic, and apostolic?" Catholics maintain that apostolicity refers to the church's being built on the foundation of the apostles, and further, that it exercises the authority that Christ gave them.
To "trace the matter," we begin with Jesus' training of the twelve and giving them authority to teach, preach, heal, and administer the sacraments. Though this authority was given to them all, it was focused more specifically on the apostle Peter. Some arguments that Catholics put forth to demonstrate Peter's primacy are: (1) Peter often speaks for the twelve in the Gospels, especially in climactic moments (Mark 8:29; Matt. 18:21; Luke 12:41; John 6:67ff); (2) Peter is often central in many dramatic Gospel accounts (Matt. 14:28-32; Luke 5:1ff; Mark 10:28; Matt. 17:24ff); (3) Peter is always named first in the Synoptic lists of disciples (Mark 3:16-19; Matt. 10:1-4; Luke 6:12-16; Acts 1:13). Sometimes the disciples are referred to as "Peter and his companions" (Mark 1:36; Luke 9:32; Mark 16:7); (4) John waited for Peter at the tomb of Jesus and allowed him to enter first (John 20:3-8); (5) Jesus singled out Peter as the shepherd of his people (John 21:15-17); (6) Paul mentions Peter first among those who saw the resurrected Christ (I Cor. 15:5). Furthermore, the book of Acts reinforces Peter's primacy among the other apostles.
Jesus had much to say about Peter's unique role as well. It was Peter that Jesus said Satan desired to sift as wheat, "but I have prayed for you," Jesus assures him. "When you have turned again, strengthen your brethren" (Luke 22:31-32). In John's Gospel, It is Peter who is told specifically to nurture and feed Jesus' sheep (21:15-19). And of course, there is Christ's famous statement that Peter is to be called rock, "for on this rock I will build my church" (Matt. 16:18-19, both occurrances of "rock" are identical [kepha] in the Aramaic which Jesus spoke).
As Protestants, there is nothing we really need to fear by acknowledging this evidence as weighty. But as we "trace the matter" further and discuss whether Peter had a successor who had a successor, then the (ahem) you-know-what starts to hit the you-know-what, so hang on to your seats....