Comparison of different quantization techniques and their impact on inference time. And how to optimize inference time with quantizations using vllm.

Local LLMs-Series: In this part I will show a comparison of different benchmarks with quantized models. The results of official benchmarks differ alot from custom benchmarks.

Azure's documentation is unclear about whether the "strict" mode is currently supported. According to the blog, it has been supported since June 2023 😅. Last week, everything was fine, but now the current strict mode leads to a generic 500 error code.

In-depth exploration of common pitfalls when working with open-source Large Language Models. Covers topics like managing expectations, selecting the right model, optimizing resource use, setting up prompt templates, and maximizing model potential.