What your neighbor did was not merely rude; it was wicked. The fear of the Lord is to hate evil, and you are not wrong to hate what happened. That kind of lewd, aggressive behavior should be taken seriously. Reporting it was right. Do not let anyone minimize it.
At the same time, do not let the fear of that man become the ruling force in your home. The fear of the Lord is the beginning of wisdom. It teaches you to hate evil, but it also teaches you that you do not have to be afraid of the wicked. The Lord is your light and your salvation; whom shall you fear? Your neighbor wants to make you feel unsafe and ashamed. If you make decisions mainly from that intimidation, you are letting him set the terms.
The question is not simply whether you should move or whether your husband should press charges. The question is what the Lord is asking of you both, together. Moving is not automatically wrong. Protecting your household is serious. But moving because a wicked man has frightened you is different from moving because you have sought God and have good reason to believe it is wise. The same is true of pursuing action. There is a place for the law to restrain evil, and using it soberly is not unpeaceable. But if the desire comes from wounded pride or revenge, it will only add fuel. The wisdom from above is peaceable, but it also refuses to call evil good.
You want this to calm down, and that is understandable. But real peace is not saying peace, peace when there is no peace. Ignoring what happened or living in silence because you fear escalation will not produce the peace you want. God's peace comes when you put the whole matter in His hands and say, Lord, it is Yours. He gives peace to the heart that keeps its eyes on Him and refuses to be driven by panic.
Do not let this come between you and your husband. You need each other. He needs your caution; you need his concern. Pray together. Let the Lord be your confidence, and do not be wise in your own eyes. The wicked may bluster, but the Lord keeps the foot of His own from being taken. His expectation will come to nothing; your hope in Him will not be put to shame.
I will pray that God gives you both wisdom, protection, and a quiet spirit. Pray for your neighbor as well, not that his sin be blessed, but that he would turn from it and fear God. Until then, keep firm boundaries, keep no fellowship with that darkness, and leave the outcome in the hands of the One who sees all things.