I'm going to tell you from personal experience that this will be a problem if not addressed. Let me give you the back story...
In 2009 we were moving back to NJ and buying our first house. The property we were buying was from an older man who was the original owner of the house. Years ago, his brother had owned the property next door which originally had a gravel driveway which, unbeknownst to either of them, actually crossed the property line. Over the years, the brother had the driveway paved and eventually sold the house to a couple. That couple eventually did things like install an underground electric dog fence around what they must've assumed was their property, but in actuality the encroaching driveway and dog fence came across the property boundary by about 3 feet.
The man who was selling the house apparently didn't even know until we had finalized a contract and the surveyor came out. At that point, it became an issue and jeopardized our entire deal as the neighbor couple insisted it was now their property essentially under "squatter's rights." There was a lot of back and forth with the seller threatening to sue the neighbors for jeopardizing his property sale to us and the neighbors threatening to sue him for the property.
At some expense to me (since my attorney had to get involved), I stepped in and offered a contract to the neighbor stating that as long as they lived in their house and I lived in mine, I would allow them to continue using their driveway without having to blow it up even though it encroached on my property. It also had some stipulation that they were responsible for their dog and its behavior as I was afraid of some bizarre situation where their dog bit someone but I would be responsible since it was on my property. Finally, in the event that either they or us moved, the driveway would have to be removed to restore the proper boundary (at their expense). Also, any fees that were incurred would be their responsibility as well.
In return for this, we asked for no money. I thought it was the neighborly thing to do so that they didn't have to remove their dog fence or blow up their driveway. In exchange, they would forego any further future BS claims that they owned a piece of my property. Sure, I would have a little less property to use and technically I'd be paying taxes for something they were using. But we thought it was the simple solution and we'd already wasted enough time and money.
In 2015 we had to move to another part of Jersey for my job. We gave the neighbors 3 months advanced notice as we had contractually agreed. They initially said nothing and we assumed that they would take care of the driveway quickly as we had agreed upon. In the meantime, we listed our house and it went under contract. A month before the closing, the neighbors came back and said that they weren't planning on honoring the contract. They complained about everything ranging from they had gotten bad advice from their lawyer to how inconvenient/expensive it would be if they had to modify their driveway. They threatened us with legal action. We tried to resolve it with them but the best they were willing to do was to help pay whatever fees were necessary to redraw the boundary line so that they could annex the encroaching property and we would lose it. They even went so far as to investigate that (even when we said No) only to find that the town doesn't allow people to do that.
Ultimately, they dragged their feet so long that we had to involve our attorney. We had to disclose what was happening to our first buyer who then backed out. We then had to find a second buyer who was fully informed about the situation, knew about the contract (and felt comfortable that it would be resolved), and was willing to buy our property anyway. However, the neighbors continued to do nothing and had their lawyer keep stalling up to a couple days before our closing with the 2nd buyer. At that point, the 2nd buyer then decided to try to back out (breaking their contract with us) since they didn't want to live next to angry neighbors. My attorney then had to threaten their attorney. Literally, the day before the scheduled closing, the neighbors finally had someone come remove the portion of the driveway that was encroaching.
We were finally able to close, but only after incurring about $12000 worth of additional attorney fees that we had to pay to fight the neighbor and the buyers. We also had to delay our closing to find the 2nd buyer. We went to the neighbors to try to recoup at least some of that. You know how much we got? Nothing... They told us we could sue them if we wanted to but they weren't paying a penny.
I will never get into a situation like this again. It doesn't matter if you have a contract that you think will protect you. There will be an attorney that can be hired to make an argument that your contract can be voided. You could also be liable if any laws or injuries happen on that property even if it's a result of their behavior. Do yourself a favor and take care of it now. No good deed goes unpunished.