Not just URL's. Google's AI snippet at the top of a search result page recently fed me the wrong phone number for Starlink support. It connected me to a scam being run at scale out of an overseas call center.
After questions like "Is your router plugged in", "What lights are on", etc., they eventually asked for my account email address and password (I was already getting suspicious by that point and of course did not provide). I could hear other agents in the background having similar conversations with other callers.
Based on some probing questions I asked, I think the agents themselves didn't even realize they weren't really working for Starlink.
I was in a hurry and should have double checked the number before calling, rather than blindly trusting the AI shoveled slop.
I tried but wasn't able to recreate the search query that produced the result, and a Google search for the number itself came up blank. A few weeks later, the number just rings with no answer.