Waste of time, is the first thing that may come in mind on thinking about broken links. A person has to put efforts into searching etc, for what is needed, and if what is seen, is a broken link, then it is a waste of time for the user who has visited the website.
I will exaggerate a bit to make things clearer. What will you do if you constantly get broken links from a Search Engine Result’s page all the way up to the 2nd page on looking for something that you need? I will obviously look for another search engine, if I find such happening even after changing the query multiple times!!! What else would you do? I guess not any different.
So, we have our answer over here. From a Search Engine’s perspective returning broken links are always going to be a strict no no. Having a website with lot of broken links are going to hamper the SE’s reputation. Obviously it will push such results below and provide its users better websites.
Loosing a visitor is a loss of reputation for the search engine. Things are not going to be any different for other websites. So, yes, broken links will cause loss of reputation for the search engines.
From a high ranking perspective, good search engines will push the websites with broken links down. It will effect reputation of websites towards the negative side.