From my experience with finetuning Gemma 12B, I noticed that 4-bit quantization would hurt instruction-following especially when it comes to NLP.