I think perhaps one of the big things that's taken me a while to really start to understand is what it meant for Jesus to be crucified for our sins. To modern sensibilities, such might appear outdated because it harkens back to the archaic practice of animal sacrifice, but that simply misunderstands the significance of that early practice.
The way I currently understand it, animal sacrifice was an attempt to purify oneself and atone for that which cannot be atoned. A person who does bad by their neighbor might be able to simply make up for their actions, but oftentimes this is not possible, and in the eyes of God, to let such wrongs go unrighted would be to abandon people to the evils their neighbors inflict upon them. Reflecting on the nature of animal sacrifice may shed light on why this in particular was chosen to atone for sin, but the central reason it suffices is that God ordained it to be so, as it is by none other than God's grace that we be pardoned.
But of course, for all its reasons, such a sacrifice is still imperfect. Can a person really push their sins onto an animal? It seems the problem here was that by letting an animal stand in for one's sins, one is only purified at best in one's animal nature. The root of sin, residing in the corrupted will, was left untreated, and sin would be recreated anew.
With Jesus as the new sacrifice, I think this points to a very direct truth about what it means to sin. And I think it can be seen as well in Jesus's teachings, like to turn the other cheek. For in order to break the cycle of sin, a person must be willing to respond to insult with kindness. Simply thinking God blithely pardons this or that misunderstands what one is asking of God to be pardoned by Him, as sin is an injustice against God Himself. So while the Old Covenant treats sacrifice as simply God's law by which He pardons, the New Covenant elevates this to the thing itself, namely what God is doing to allow us to be reconciled to Him despite our sin.
And I think this somewhat sheds light on the misunderstanding that Islam has of vicarious atonement. Because while repentance is the correct response to sin (something Christianity and Islam can agree on), one must bear in mind that sin always comes with a very real price. To say otherwise is to treat sin as an arbitrary law, and to deny the true evil inherent in sin.
As usual, thoughts are welcoming. Still learning and stuff.