Your intentions play a bigger role here.
If your intention was to purposefully upset someone, then the guilt within which will create the resulting suffering is your punishment in itself.
If your intention wasn’t so, then eventually your actions will prove so and you will become free from that thought.
If you are free that means your actions are aligned with the reality and your mindset and though process are clear.
Karma doesn’t mean tit for tat. It starts its work as soon as you commit some good or bad deed. Its a very internal thing. There is no one sitting and counting as to what deeds you are committing.
Therefore, every action of yours matters. Stay aware and give a thought before doing anything.