If you have FTTH, do they allow you to use your own router? I know this is possible with Verizon FIOS. See if bypassing the ISP's router with one of your own resolves the issue.
Have you tried asking the ISP if they pushed any firmware or malware updates to the Internet Modem recently? On a related note, try having them completely reset the settings in the Internet Modem to whatever is considered factory.
Another thing I like to do is use
PingInfoView to run ping tests. It's a Windows only application. You can ping multiple addresses, and log the results. Generally, I ping another computer device on the LAN to see if the issue is local. I also usually ping the router (Internet Modem). Finally, I ping a couple locations such as Google's DNS (8.8.8.8) and Cloudflare's DNS (1.1.1.1). I set ping times to about 5s intervals. Walk away, come back in a day or so, and review the results. I usually restart the test and save the results to a new log file every new day.
For a general test of this nature, if the ping times on the LAN are good, and only ping times to the Internet is high latency, then that would be an Internet issue, and not local.
If all ping times suck, local and Internet, then it's something on your end.
I've seen some crazy stuff with network devices. Something as simple as a crappy IP camera has screwed up network printing and even some business critical network hardware on some of our client's computers.