Hacker News

points by gophigure 4 days ago | 0 comments

What I find problematic, especially for teams that rely on GitOps and GitHub being (relatively) stable, is the fact that GitHub's status page can't be relied on. You arrive at the conclusion, "this must be GitHub," but the status page rejects the notion, and so you go digging. That can lead to a lot of toil and wasted time, in addition to sideways looks paired with comments like, "but status shows green."

A sick person with symptoms is clearly sick, but sometimes they're asymptomatic. That's what the GitHub status page has become. When it's red and showing problems, you know you have problems. When it's all green? You know you MIGHT have problems.

That can make outages like this harder to triage, since reality is a black box.

[–] ozyschmozy 3 days ago

> "this must be GitHub," but the status page rejects the notion, and so you go digging

Don't know about you but at this point I've learned to recognize and ignore GitHub issues even when they haven't admitted to it yet

[–] arccy 4 days ago

This is why you ignore everything for at least 30 min before starting to investigate...

[–] chrisjj 3 days ago

Each status page update requires preapproval by the PR dept.

[–] defraudbah 3 days ago

show me one status page that's reliable, it turned into smoke and mirrors

[–] solyin 3 days ago

downforeveryoneorjustme.com is a high level way to gather that data

[–] thwarted 3 days ago

Extremely high level, and mainly merely "is the site accessible". However, github having issues with "issues" or "pull requests" is a functionality, not an accessibility, degradation, which can often only be discovered by actually trying to perform the action/use the functionality and comparing the result to he expected result.

[–] jeroenhd 4 days ago

Github is fast in updating their status compared to other companies. Amazon has to be bombed, on fire, and in the news before their status page shows anything about any ongoing issues.

I assume these companies have some kind of uptime guarantee in their contracts, because they seems to care more about covering their asses than about informing customers.

[–] sleepybrett 3 days ago

No they aren't, we can see outages coming 20+ minutes before GH reports them.

We have outages on our books that they never reported even after we ticketed them.

They cook the stats like an inner city police department.

[–] beardedetim 3 days ago

Me and a colleague were just ranting about how there's no way in hell Actions has 99.6% uptime but yet that's what their status page says. Maybe it's just bias and I am only remembering the minutes that it's down what feels like every week or multiple times a week. But I'm suspicious of their stats, even if they declare an incident on the status page.

[–] sleepybrett 2 days ago

I would suggest every enterprise customer that is not keeping track of github downtime and it's effect on their systems and workforce independently for presentation to their sales team every billing cycle for discounts is leaving a lot of money on the table.

[–] chanux 3 days ago

Not fraud from a legal point of view I guess /s

[–] deathanatos 3 days ago

> Github is fast in updating their status

Github's status page is incredibly slow in comparison to the status page Github themselves used to have, which had several key metrics of various systems that would update in real time, and when they did have a (comparatively rare) outage, would usually have at least one metric going "yeah, something is wrong".

[–] giwook 4 days ago

Comparing one shitty set of standards to another shitty set of standards… hmm

[–] spartanatreyu 3 days ago

What you're looking for:

Question 1: Is it github, or is it everyone:

Answer: https://downdetector.com/

(If everyone's having issues, then stop. Otherwise, continue below...)

---

Question 2: Is it github?

Answer 2: https://www.githubstatus.com/

Question 3: Why are you lying to me, is it github or not?

Answer 3: https://mrshu.github.io/github-statuses/

(If github is not having issues, then check your local environment. Otherwise, continue...)

---

Question 4: Why is it always github, surely they're not that bad right?

Answer: https://www.dayswithoutgithubincident.com/

(Perhaps its time to consider an alternative to github, or at least a mirror)

[–] c0rruptbytes 4 days ago

I can imagine a lot of people moving to OCI instead of GitOps

[–] stryan 4 days ago

OCI layering on top of running systems a la bootc? or OCI as the desired state storage for CM?

[–] stackskipton 4 days ago

SRE here, OCI for storing deployment config information, most of time, Kubernetes manifest files/Helm Charts.

[–] stryan 4 days ago

Gotcha, that's what I figured but I was still curious. I've got a side-project that was originally designed for traditional GitOps but got OCI support as a feature submission from a user. I've been seeing more usage of the OCI backend these days and originally figured it was a side-effect of the project domain (Podman Quadlets) but maybe it is an over all field shift.

[–] bdsa 3 days ago

In the last couple of weeks, I've been moving a deployment stack off ArgoCD pointing at Git, and ended up settling on Flux fetching OCI artifacts from ECR. The artifacts are built and pushed by GitHub Actions. It seems like a nice way to "have GitOps" without needing GitHub to be up all the time.

[–] c0rruptbytes 3 days ago

I think so, Argo CD appsets are even getting an OCI generator in the next release (3.6)

ECR just seems to work really well for most people and IAM is nice compared to Github Apps + rate limits