The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
[–] amluto 2 days ago
> $ret = crypt($pw, $hash); $ret == $hash
musl is actually safe against this use. OpenSSL takes a bizarre code path, detects that $hash is too short, calls crypt($pw, $other_string), forgets that it detected that the hash was too short, and checks $hash == return value.
See my other comment.
[–] mpyne 2 days ago
[–] __david__ 2 days ago
[–] idlewords 2 days ago
[–] markasoftware 2 days ago
[–] numeri 2 days ago
The surface-level hints are gone, but the attempt to make every sentence pithy, the decisions on what information to present and what to assume the audience knows, and the over-emotionalization are all still there.
[–] lukax 2 days ago
[–] ghostly_s 2 days ago
[–] larsonian 2 days ago
[–] alexgoodhart 2 days ago
[–] pizzafeelsright 2 days ago
In the past I could get an explanation, where I was able to track the line of thought. Now when I read through the explanation it is like someone has taken two or three explanations and merged them into each other, like there is a combining of iterations without clarity.
example:
The tree is blue.
The tall tree has the color of blue.
the tree, while tall, is blue.
That finally gets squished into: the blue tall tree has height and is evident of blue.
Is this double speak or injected double effort of words?
[–] endymi0n 2 days ago
[–] ericbarrett 2 days ago
[–] yearolinuxdsktp 2 days ago
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
[–] xnorswap 2 days ago
< mundane detail>
< If you don't run this command your computer will explode>
< exposition >
< mundane reminder to follow up something else >
Then 3 back and forths later, your computer explodes, and claude has the cheek to point out when questioned that it did warn you.[–] brabel 2 days ago
[–] phyzome 2 days ago
Except it turns out that at least one of those things is because I didn't enable JS, and the static text is wrong!
(Turning on JS did not actually help, because then the page is filled with horrible, cheesy animations and I can't actually read any of it.)
[–] byronsharman 2 days ago
[–] dboreham 2 days ago
wtaf ?
[–] za_creature 2 days ago
[–] mmoll 2 days ago
[–] zbentley 2 days ago
musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
That was a misfeature and should never have been implemented. But I bet it was an easy band-aid change to make!
[–] ghostly_s 2 days ago
[–] pdw 2 days ago
> Upon error, crypt_r, crypt_rn, and crypt_ra write an invalid hash to the _output_ field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
[–] crunchatized 12 hours ago
And that's also only in Debian 11 and later. That man page said something entirely different in Debian 10 and earlier, when they shipped glibc's libcrypt. Now they ship libxcrypt instead.
Fedora asked libxcrypt to give them back glibc's behavior, returning NULL, and they obliged by adding ./configure --disable-failure-tokens. But it's an option that Debian never enabled.
https://github.com/besser82/libxcrypt/issues/25
https://salsa.debian.org/md/libxcrypt/-/blob/master/debian/r...
[–] cloudfudge 2 days ago
[–] bryan_w 2 days ago
That seems orchestrated.
[–] bryan_w 2 days ago
https://lists.llvm.org/pipermail/llvm-dev/2019-June/133308.h...
[–] bch 2 days ago
Of all the things to mitigate...
1) crashing should be an option - it noisily indicates "there's a problem" and if it faults and dumps core, you've got a guided tour of your problem.
2) Is testing NULL as a passable option likely? I'd guess a more likely idiom is:
if(NULL==(str=somefunc(arg1, arg2))){
return err;
}[–] xp84 2 days ago
Wise words. I think early in my career I tended towards coding as defensively[1] as possible in a very specific way of "don't crash", as I definitely was not working in codebases I felt I had a full picture of. "I don't know who calls this method, and I don't want to get a call in the middle of the night that I broke prod" so I would say, return an empty string when asked to format a datetime except I'm passed a string or an int. (Yes, I was working in dynamic, untyped languages, too).
Later on, even in untyped languages, I came around more to "Throw loud and specific exceptions when asked to do things that are nonsensical" - it requires more overall cojones though because it may cause "your code" to crash. But the realization I came to was that it's better to crash ('a MaliciouslyLongPasswordError was uncaught') because "AuthManager asked PasswordService to check a password that was 2MB long" than to paper that over until something like this comes out. Or, the more common situation, you're passed garbage, you return something that won't choke the caller at the time, but what it saves to the database in that scenario ends up being corrupt data. 6 months later someone notices, and has to sort out a data mess that may or may not be recoverable.
[1] note that most "code defensively" advice is good. What I was doing could be thought of as a foolish misinterpretation of the phrase.
[–] oasisaimlessly 1 day ago
[–] zbentley 2 days ago
I dig it.
[–] PokestarFan 2 days ago
[–] creata 2 days ago
> The behavior of crypt on errors isn't well standardized. Some implementations simply can't fail (except by crashing the program), others return a null pointer or a fixed string. Most implementations don't set errno, but some do. POSIX specifies returning a null pointer and setting errno, but it defines only one possible error, ENOSYS, in the case where crypt is not supported at all.
In my opinion the problem is that the behavior of crypt is underspecified - it isn't even specified how it signals failure, which is shocking.
Another disturbing decision that crypt's signature takes in the hash its output is supposed to be compared against - if it weren't for that, this half-baked "on error the output won't compare equal to the input" measure that crypt and crypt_* use wouldn't have been possible.
You can't check errno to see if there was a failure because, as the man page says, the value of errno is unspecified when crypt succeeds.
[–] throwaway2037 2 days ago
> You can't check errno to see if there was a failure because, as the man page says, the value of errno is unspecified when crypt succeeds.
You can set errno to zero before the call. If the value is non-zero after the call, that is a good heuristic for failure.[–] za_creature 2 days ago
Why * specifically? Why not some random guid generated before committing, if it's so important to not let buggy software crash?
[–] zbentley 2 days ago
But buggy software might not crash; it might compare NULL == NULL and misbehave, instead.
So there's really no good option here, which is why C is a rancid choice of language for publicly-usable APIs. But we're stuck with it. So it goes.
[–] dboreham 2 days ago
[–] zbentley 2 days ago
But issues caused by folks forgetting to check errno, miscomparing NULL to NULL when calling multiple APIs, dereferencing external-API-returned NULL due to missed error checking, mishandling inherited errno after successful action, or--the most common--mishandling negative integer return sentinels meant to indicate errors? Those are incredibly common.
[–] tredre3 2 days ago
If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not
be able to use a unix password to log in (but the user may log in the system by other means).[–] cowlevel 2 days ago
[–] nextaccountic 1 day ago
[–] dboreham 2 days ago
[–] JdeBP 2 days ago
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
* https://cgit.freebsd.org/src/tree/lib/libpam/modules/pam_uni...
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
* https://cvsweb.openbsd.org/annotate/src/lib/libcrypto/des/At...
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
[–] boomlinde 2 days ago
No. https://man7.org/linux/man-pages/man3/crypt.3.html
My guess is that you are reading the libxcrypt crypt(3) manual: https://www.mankier.com/3/crypt
That's not GNU libc, but frequently deployed on Linux systems.
[–] JdeBP 2 days ago
* https://manpages.debian.org/trixie/libcrypt-dev/crypt.3.en.h...
Although that's a fine distinction for the point being made, as it's still the manual page for the crypt() function that one gets on the operating system, which that other manual page is not.
[–] boomlinde 1 day ago
[–] JdeBP 1 day ago
* https://sourceware.org/git/?p=glibc.git;a=commit;h=e6e3c6668...
Notice that Debian didn't list your manual page, which doesn't accompany a crypt() function at all, since it isn't actually the GNU libc manual but a separate independent manual for Linux. The GNU libc manual nowadays acknowledges Michael Glad's work but no longer documents crypt(). It used to, and it used to explain some of the 'impossible' encrypted string values:
* https://sourceware.org/glibc/manual/2.38/html_node/Passphras...
You noticed that I'd not got the GNU libc manual. Neither had you. (-:
[–] ruslan 2 days ago
Damn, that's a bug! Please file bug report.
[–] JdeBP 2 days ago
* https://cgit.freebsd.org/src/tree/lib/libcrypt
As I said, the callers of crypt() have for a long time been using the knowledge that crypt() on their platform simply never returns NULL. Unix has never been returning NULL from crypt() as far back as 7th Edition, when it simply always returned a pointer to a single internal static buffer.
* https://github.com/dspinellis/unix-history-repo/blob/Researc...
The reality is that all C library people have to work with a corpus of Unix applications that do not check for crypt() returning NULL that goes back almost 50 years, and an error return mechanism that barely exists outwith the pages of the SUS.
That includes callers in single-user mode in process 1 in some configurations.
* https://cvsweb.netbsd.org/bsdweb.cgi/src/sbin/init/init.c?an...
[–] ruslan 2 days ago
* https://cgit.freebsd.org/src/tree/secure/lib/libcrypt/crypt-...
Failing to check result of crypt() in init cannot do much harm, doing so is equivalent of going system reboot (crash init :-)). Besides, whoever has access to system console can do whatever they want with the system, no password can prevent from that.
[–] JdeBP 2 days ago
* https://man.freebsd.org/cgi/man.cgi?query=crypt&sektion=3&fo...
NetBSD's manual is more detailed on the subject, and explains the bind that library implementors have been in for decades, thanks to almost 50 years of applications knowing what crypt() 'really' does.
* https://man.netbsd.org/crypt.3#RETURN%20VALUES
And this despite the SUS trying to shoehorn an error return mechanism into crypt(3) in order to accommodate the problems that operating systems had exporting encryption code, two decades afterwards. (The 1997 GoSolo book explains that this is why there's an ENOSYS error.)
[–] za_creature 2 days ago
There is simply no universe in which a security-conscientious reviewer looks at the code, spots * being a valid return value, remembers that * is the default value in /etc/shadow for "no password login please" and then is all like:
This is fine, you check for * and return + instead.
There are more 32-long character sequences than atoms in the universe. Pick one, any of them instead.
[–] odo1242 2 days ago
[–] mmoll 2 days ago
[–] tredre3 2 days ago
Of course given the chance that one might do a direct string comparison between crypt's return and the passwd/shadow file's hash field it was still a not well thought out choice.
[–] gf000 2 days ago
[–] dboreham 2 days ago
[–] 1vuio0pswjnm7 2 days ago
OpenSSL??
[–] mpyne 2 days ago
[–] smnc 2 days ago
A typo: `$ret = crypt($pw, $salt)`
[–] paparazzi678 2 days ago
[–] forrestthewoods 2 days ago
[–] Woodi 2 days ago
Not to programmatically parse 'ls' output or some /proc/some/file.
Eg. git started as Perl scripts then moved to C to realy have data straight from C calls from system.
So for example all Internet things calling "openssl -some -options ...." are pure trash :>>
[–] chuckadams 2 days ago
[–] markasoftware 2 days ago
[–] GuB-42 2 days ago
It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer. C++ has references in an attempt to fix this problem.
Because C tends to prioritize performance over safety, a NULL pointer dereference is considered undefined behavior and the compiler can invoke nasal demons if it makes the code go faster, but with the right options, you can have it checked and invoke what in other languages would call a panic handler, but on UNIX is the SIGSEGV signal. You can then attach a debugger or read the core file and see what went wrong.
Note: I am not talking about "char *" for strings, that one is unsafe, but NULL is not. It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.
[–] LordDragonfang 2 days ago
I had never heard that phrase, and was delighted at the answer when I googled it its origin, thank you:
http://www.catb.org/jargon/html/N/nasal-demons.html
> “When the compiler encounters [a given undefined construct] it is legal for it to make demons fly out of your nose” (the implication is that the compiler may choose any arbitrarily bizarre way to interpret the code without violating the ANSI C standard)
[–] cowlevel 2 days ago
I've heard nasal demons invoked as a literal possibility. No they are not a literal possibility.
[–] Xirdus 2 days ago
[–] cowlevel 2 days ago
Which is zero, to be clear. There's not even a standard USB device class for a nasal demon releaser
A null check elision is not a demon up my nose - it's a null check elision.
[–] miki123211 1 day ago
That's... a bizarre choice, considering how religious C is about not making any assumptions about what a machine can and cannot do, even going to the trouble of working with non-IEEE floats and int types that are not a multiple of 8 bits.
[–] annex-winged-cr 1 day ago
You don't even need a fancy result type. A simple Option<String> would've forced downstream users to consider the failure case, avoiding this situation in the first place.
[–] kazinator 1 day ago
Someone coded a defense against long inputs which ignored this important principle.
What turned the bug into a security disaster is that Musl stupidly chose two symbols such that one of them is exactly the same as a convention for disabled logins. So that the moment someone makes a change to the function which allows crypt("something", "*") to return "*", you have an incident.
If the same bug were made in Glibc, it would have limited impact because crypt("long .... string", "*0") returning "*0", while incorrect, wouldn't do anything in a system that doesn't use *0 in password files. *0 does not correspond to any widespread password field convention.
[–] f33d5173 1 day ago
Now what?
[–] ahepp 1 day ago
Whereas if the library returns something that is unambiguously an option type, consumers unambiguously call unwrap at their own risk.
Of course you do have a point (assuming I'm picking up what you're suggesting correctly) that all pointers in C are option types. But I think there's a cognitive ergonomics aspect to this where an option type more clearly relays the library authors' intent.
[–] shakow 1 day ago
[–] f33d5173 1 day ago
[–] kazinator 1 day ago
[–] eviks 17 hours ago
[–] miki123211 1 day ago
If NASA were to use the equivalent of `pub struct Newtons(f32); pub struct PoundsForce(f32);`, where the types Newton and PoundsForce were not interchangeable, this would never have happened[1].
[1] I'm aware that that story is a bit more complicated than that, that CSV files were involved (which are, again, not type safe), and that, in that particular scenario, more would have to be done to preserve type safety across formats, vendors and codebases, but my point still stands.
[–] gugagore 1 day ago
https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-...
> The newtype is opaque, so the only way to observe it is to convert it back to a [f32].
[–] poly2it 2 days ago
[–] eru 2 days ago
That's a bit weird? In Rust, if you have an thing that returns Result<String, String> that's still type safe, even though success and error types are the same.
I think I understand what you want to express, maybe. You might say it's in-band vs out-of-band error signalling?
[–] itishappy 2 days ago
These types are not the same:
[–] markasoftware 2 days ago
[–] kazinator 1 day ago
If you have literally the above convention where a valid result is a String, and an error result is also a String (e.g. error message) that is still type safe.
The success type and error type having to be different is not a good way to express the idea; rather that success and error cases are distinguished by type (even if the success and error data units are of the same type).In other words, the "Ok" and "Err" entities are an ingredient of the type and that effectively makes the result and error be distinguished at the type level.
But we could also argue that Rust still allows you to write an integer-returning function such that -1 means error.
[–] eru 1 day ago
Of course that breaks down, when the success type is also an error message, eg when you are looking up something in a catalogue of error messages.
Rust has no trouble with this, for exactly the reasons you line out.
[–] quotemstr 2 days ago
[–] poly2it 2 days ago
[–] pron 2 days ago
[–] ciefa 2 days ago
[–] quotemstr 2 days ago
You don't have to think C is sound or good to distinguish a disfavored API style from mathematical unsoundness. There are infinitely many type safe APIs and languages that are, despite their soundness, ugly and hard to use.
Let's not project every dimension along which a language can be good or bad onto the axis of safety.
[–] majorchord 2 days ago
[–] alerighi 1 day ago
However a better API (to me) that I would have written was:
That way btw the function doesn't need to allocate memory inside, which is (being a mostly embedded programmer) something I don't like at all. Plus returning an error enum makes it clear instead of having to look at errno.But POSIX API is not something well designed, it dates back to the '80 so... probably today would be written completely differently
[–] kazinator 1 day ago
It is not immune to the present issue. If our reentrant crypt is given a long password and the salt "", and it decides this is an error, placing the output "" into the result buffer, then it has the problem. It has the problem in a thread-safe way, but it has the problem. :)
[–] itishappy 2 days ago
These are not the same, and your next sentence suggests why. Option has but a single error type, but result types and error unions allow arbitrary types of error.
[–] SkiFire13 2 days ago
That depends on how you interpret the pointer type, but in any case there are multiple interpretation and just the fact that these exist will create confusion and type unsafety.
[–] rjbwork 2 days ago
"Make invalid states unrepresentable" is such a useful adage when designing even basic systems, much less the gates to control your server.
[–] citrin_ru 2 days ago
[–] shess 2 days ago
[–] jeremyjh 2 days ago
[–] creata 2 days ago
Maybe I'm wrong, but it looks to me like the exact same incentives and means to screw things up exist in C and in Rust for this particular bug.
[–] markasoftware 2 days ago
For example, in a Rust web app, you can choose to use String for encoded URLs, unencoded URL fragments, escaped html, unescaped HTML, usernames, etc. Or you can choose to use newtypes and carefully design all APIs to only take and return the correct newtypes.
[–] gf000 2 days ago
The problem is not that you can write bad stuff in rust, the problem is that this particular usage pattern can't be sanely coded in C. This is a public API that should inform users and enforce certain properties. A Results type does exactly that, you would have to go way out of your way to force invalid stuff through that type alone.
[–] creata 2 days ago
[–] cowlevel 2 days ago
[–] speedstyle 1 day ago
But in your version, you can't preserve backwards compatibility, making an infallible function fallible is a breaking change. For something comparable to libc you might want to (eg Rust std still has mem::uninitialized), but even with security issues, I feel like people would be more inclined to deprecate for a new function (or panic, which does have its own problems).
A good type system also means things are written with better forwards compatibility, and can be safely changed later. The Rust equivalent to `crypt` likely wouldn't exist: a low-level library would just provide hash functions, leaving callers to mitigate DOS etc; a broader library might return some opaque handle with equality/serialization, which you couldn't try to compare to a passwordless user; a high-level library would parse the account store itself. None of these really need to change in incompatible ways, they have all the information to add features and fixes at their own level of abstraction.
[–] eru 2 days ago
But if you have a stronger and more expressive type system, you are more inclined to use it, instead of looking for silly workaround like in C.
[–] skew-aberration 2 days ago
[–] justmedep 2 days ago
[–] cowlevel 2 days ago
[–] zbentley 2 days ago
And that just ends up indicting the C language itself. Can't send errors down a separate channel; folks might not inspect errno. Can't return NULL on error, folks might compare it blindly to another result. No tagged unions, so can't return one. Can't crash on error because there's no scoped error handling. Can't return a struct for checking .error/.result separately, because alignment and equality (and tearing and accidental clobbering and more) make that brittle as hell for widely-used APIs.
C is not a good language for library APIs. Like, really not. It has its uses/strengths, sure, but without even getting into the typical gripes people have about C, libraries--one of the things that it is used for the most--are one of the things for which it is the least well-suited.
"But it has a stable ABI!" So does a land mine.
[–] cozzyd 2 days ago
[–] cowlevel 2 days ago
[–] chuckadams 1 day ago
[–] kps 1 day ago
[–] eru 2 days ago
Another example that C programmers love: using 0 to mean infinity. Eg when you have a timeout setting, and 10 means ten seconds, but 0 means infinity seconds instead of zero seconds.
[–] cowlevel 2 days ago
[–] eru 2 days ago
[–] cowlevel 1 day ago
[–] eru 1 day ago
In Haskell, the closest thing to adding 5 to an optional value would be something like:
This one leaves 'Nothing' alone, but adds 5 in case the value is present.[–] Joker_vD 2 days ago
[–] justmedep 2 days ago
[–] eru 2 days ago
What Rust calls an enum, but an enum with payloads. In C terms it would be a discriminated union.
[–] kazinator 1 day ago
Firstly, why do crypt functions return a valid string to indicate an error? The problem is that the classic null return makes some programs crash which are not expecting it.
musl has a correct protocol for the error string, similar to the *0/*1 convention. It also uses two symbols, * and +. If the salt string is *, then it uses \+ specifically to thwart this issue: the program blindly assuming that the password is correct because the output of the function matches the hashed password field.
Where it seems they screwed up is this:
I.e. in this special anti-DDoS measure that checks for a too long input, they forgot to implement the key piece of logic that the output of the function, when an error is being reported, must be different from its second argument (the "salt", which is usually the entire hashed password or the contents of that field from a password file).What we want is:
Thus the application ends up comparing "*" and "+", or "+" and "*" so the authentication fails.Under no circumstances should "*" be returned when the second argument is "*", or "+" when it is "+", because it must be assumed that the app doesn't do any sanity checking, and blindly compares the second argument to the return value.
[–] mpyne 18 hours ago
However OpenSSH passes a different hash to musl’s crypt() than what it later uses to compare, so musl doesn’t switch to the alternate return output because from its perspective, it’s already guaranteed not to accidentally compare the same as the input hash.
[–] selfmodruntime 1 day ago
[–] l33tman 2 days ago
[–] sgerenser 2 days ago