Hacker News

points by gokaygurcan 3 days ago | 0 comments

I help maintaining one of these awesome lists, awesome-javascript, since 2018. There's been always self-promoting or similar activities, or someone coming up with an idea/forking a repository and doing some small changes, then thinking they're suggesting something groundbreaking. Because of this, we needed to add a few checks over the years to the pull request template, that basically asks if the resource is out there for a while, actively maintained, actively used by people, etc.

Recently, thanks to AI being this cheap and accessible, the amount of shitty PRs quadrupled if not more. And the worst part? Agent doesn't care about the checklist at all and re-writing the PR body with bunch of nonsense that we maintainers need to read, understand, and evaluate. I started keeping the score for the most shittiest PR based on how soon after the first commit made they open the PR. It was 11h, then 10h, and now it is 6h. How on earth a 6h old repository with zero user is considered "awesome" I don't know, but people should stop prompting their agents to create 10s of these PRs every day.

Yes, there's always an alternative: stop maintaining these lists. I don't think it's a viable option either. Sometimes you start using a new piece of tech and want to have access to somewhat curated list of resources that might speed up your understanding. There's still a bit of value to keep them alive and it's dying out bit by bit (literally) every time someone mass-creates these spammy PRs.

Thanks for reading my unsolicited rant.

[–] LelouBil 3 days ago

In the age of unlimited and very fast content production (for any media, code, images, music, text, or otherwise), I think content curation like this may be more desired than ever.

Maybe I'm too optimistic, but personally it would lower the burden on me so much to have a curated list of *awesome* projects.

The fact that another person has taken from their time, so the hundreds or thousands of people reading this list don't have to do the same amount of investigation to figure out if a project is truly awesome, is in itself awesome.

[–] opem 2 days ago

Exactly good awesome lists have always been and will be the "goldmine" of curated resources. Not only these, I see all slow paced thoughtfully put in content as such. Like take StackOverflow for example. But it's really unfortunate to see people give up.

[–] NichoPaolucci 1 day ago

Maintaining these lists USED to be a lot easier though.... Once you found a good resource, you could generally chuck it into the list and, for the most part, expect it to continue to be 'good'.

Now, good resources can turn into bad resources much more quickly, so maintenance + keeping track probably requires a lot more effort.

[–] alansaber 2 days ago

Yes indeed this article points at some flaws but none of which I think are existential, popular lists becoming outdated and unmaintained whilst also being bombarded with crap is the norm

[–] armchairhacker 3 days ago

> Agent doesn't care about the checklist at all and re-writing the PR body with bunch of nonsense that we maintainers need to read, understand, and evaluate

If it can’t follow a simple checklist, it shouldn’t be a problem, because you don’t even see the PR, because it should be auto-closed.

> How on earth a 6h old repository with zero user is considered "awesome"

It’s not, and you can write a bot that auto-closes such PRs (and any PR from an obviously AI account, and I wish pangram was cheaper, but any PR that is obviously AI text).

[–] dayjaby 2 days ago

He didn't ask if a 6h old repo is awesome or not. Of course it's not.

Ok, repo age being one of hundreds of red flags. Each of these can be auto-detected. But each auto-detection rule will be known and can be gamed afterwards.

[–] chacha102 2 days ago

Age is probably one of the metrics that could take the longest to abuse, since it directly goes against the instant-gratification that the agent-runners are looking for. Requiring the a repo be around for a couple months doesn't sound too bad.

Someone might grab an old repository and just keep force-pushing new content and thus repurposing it, but if you can find a quick check against that (probably a log somewhere), I think it has a fairly good chance of weeding out the noise.

[–] armchairhacker 1 day ago

It’s easier to make the rules than game them; ultimately you’re the arbiter, so once you recognize slop, create a new rule that filters it and nothing legitimate.

“Must have six months of active development” beats every slop I’ve seen yet (and we’ll see in six months, but vibe coding has already existed for longer). “Must appear in a popular app (by my judgement), be vouched by someone I trust, or exist before 2024” would probably accept most legitimate “awesome” repos.

[–] utopiah 2 days ago

Just to be this pedantic, every single awesome repo was at some point 6h old, and hopefully they were awesome already.

[–] gokaygurcan 2 days ago

Yes, we were all 6h old babies at some point. Some of us became great people, some of us became terrorists or worse. Takes some time to see where we end up with. No different for these resources, especially the repositories. If it's an internal tool that's heavily used, then made public with a wiped history, maybe your argument is correct but if it's a never-done-before/new approach to any problem, it needs some time to see the traction and proof.

There's also some form of outcome bias here. You're judging the resource/repository/library/whatever from today's learnings and standards, but it took this amount of time to reach here, maybe do-over multiple times, 100s or 1000s of commits and clearly none of these were there when it was only 6h old.

[–] dataflow 2 days ago

If by "hopefully they were awesome already" you mean "hopefully deserved gaining widespread traction in the first 6h", then absolutely no. Try the analogy with something physical (like an airplane) to see how ridiculous it sounds. No passenger should've gotten on $the_absolute_most_awesome_vehicle until it had passed the stress and crash tests, been certified, etc.

[–] yjftsjthsd-h 2 days ago

Were they? I'd expect that even a project that becomes "awesome" was at best "okay" or "promising" at 6h.

[–] MisterMunchkin 2 days ago

I don’t think Linux was very good 6 hours in

[–] xnx 2 days ago

I know there's been a lot of discussion around pay to send as a means to reduce email spam, but pay-to-PR (for unverified submitters) seems very reasonable.

[–] RockyMcNuts 2 days ago

So easy to make it self-curating, if not self-updating with e.g. Karpathy wiki workflow.

PRs are not a great way to maintain it, need a front end that requires a human-shaped effort, email and CATCHA, although agents will figure it out.

But then the back end can be automated to some extent, automate getting a signal as to whether someone put real effort into the linked resource, whether the resource has any traction.

At the end of the day the whole point is knowledgeable human SME curating the list, so it will be hard if it's a fast-moving topic. And that is also what makes it useful, people can leverage other people's effort.

Anyway wikipedia still seems to work but it takes a well-motivated, well-tooled army. Maybe we need that, some kind of updated wikipedia for tech updated by SMEs with the right tech stack behind it. A sort of meta-stack-overflow that tells people and agents where to go to get answers.

[–] MisterMunchkin 2 days ago

Use your own AI to aggressively close all their slop requests.

[–] rickdeckard 3 days ago

unsolicited rant read and appreciated.

(suggestions to expand it will be made in a dedicated PR /s)

Great to read that there are still people out there sifting through the noise of Agents shooting PRs in all directions.

In the end you may have to recruit an AI agent to do the (pre) review, otherwise you're ending up being the only human driver on an automated highway (place mental image of Will Smith in I-Robot here)

Keep up the great work, it's appreciated!

[–] aftbit 2 days ago

I'm reminded of the poor human moderator of that German wiki that OpenAI agents hijacked as a message board, who spent hours reverting edits and removing slop pages created by the AI.

https://collusion.wiki/#the-agents-realize-the-site-owner-is...

[–] forgotaccount3 3 days ago

> Yes, there's always an alternative: stop maintaining these lists.

Or use your own AI judge agent to filter out the submissions that clearly don't align with the intention of the list.

[–] gokaygurcan 3 days ago

I appreciate the suggestion but in this case I don't believe the solution to AI problem is using more AI. I personally believe what makes these lists "awesome" is the human element, not some deterministic checks that another agent could run.

[–] aftbit 2 days ago

IMO it's just like using an anti-spam filter in your email. An AI agent should never decide "Yes, this is awesome" but it surely can decide "No, this is obvious trash". Sometimes the simplest rules are best - "Did they follow the PR template? No? Ok, auto-close with a message saying that AI submissions are not allowed"

[–] NateEag 2 days ago

You don't need an agent to check if the PR matches the template.

[–] fn-mote 2 days ago

The point is: it’s easier on the humans if submissions not following the template are auto-rejected.

Or do you mean just grep for “TEMPLATE-1.3” and if it’s not there, reject?

[–] NateEag 1 day ago

Yes - a deterministic tool is just fine for verifying that a template has been followed.

It's not quite as simple as "check for one line," but it's not hard to have a useful check that costs far fewer resources to run than an agent would.

[–] Roark66 2 days ago

The deterministic check could help you filter out the chaff - prevent the human element from getting DOSed.

[–] orthogonal_cube 2 days ago

That assumes one has the money to maintain an AI service/subscription, resources to run the prompt from, and the time to fine-tune the AI to be dependable with a high rate of accuracy in your criteria.

Maybe someone who has a lot of free time and high enthusiasm for AI might consider this to be worth chasing after, but anyone who has dealt with these whack-a-mole scenarios is going to admit the easiest thing to do is just not maintain the resource anymore. There are less-draining projects and endeavors to turn towards instead.

[–] dasil003 2 days ago

No. Judgment is the thing AI lacks the most. Sure, you can try to give it guidelines, but it sucks at interpreting them because it has no taste and no common sense. It is trivially influenced by bullshit submissions because it overindexes on text it sees and has no theory of the mind or sense of intent.

It is like a naive and prodigious intern with incredible intelligence and factual knowledge, but no sense of how the world works or what will actually fly with human stakeholders.

[–] bunderbunder 3 days ago

Sounds like they don’t even need to pay for AI. Maybe a deterministic script that does some basic quality checks like verifying that the linked repository isn’t day old slop, and auto-rejects the ones that don’t pass muster would be better. For starters, then you could have well-established and documented rules. Which people generally seem to prefer over AI referees that reject things for opaque and confusing reasons (e.g., YouTube moderation) and can’t be reasoned with.