Obviously, I have no clue about how an LLM works. Having said that, I cannot fathom how synthetic data could improve things much. My suspicion is that it gets favored since it is so predictable to generate, while finding and filtering actual information is hard and messy.
I just tried to generate the most simple code for Apple Metal — it never worked. Looks like GPT could use a closer look at actual example code from that area. And, like that, there are probably countless other aspects of what humans have come up with so far that warrant a closer look.
“Organizing the world’s information and making it accessible”
That is still a good guideline, as is “Don’t be evil.”
Their mileage might vary.