Machine Learning

I recently read Hamel Husain and Shreya Shankar’s article about similarity metrics. Their argument is that metrics like ROUGE and BERTScore often miss the failures that matter in an LLM application. I agree. But I think we need to separate two things: evaluating what a model learns during training and evaluating whether a RAG application works. Both involve an LLM, but the questions are different. If you do not understand the question, choosing a metric will not help you much. ...

1. From Jev to Laya I recently read TypeSafe’s announcement of Jev. The idea is interesting: give a model some information and predefined questions, then get structured decisions with probabilities. A support workflow, for example, could use it to choose which team should handle a ticket. But a structured answer can still be wrong. I wanted to see how this kind of model performed on a concrete task. Then I found Laya, an open source decision model. Its author, Nandakishor M, claims he worked on this direction “a year ago.” His March 2025 SalesRLAgent paper documents earlier sales-prediction research. His write-up describes developing the general-purpose model afterward, so I would not say today’s Laya checkpoint was released a year ago. ...

Running large language models in production is expensive. A single Llama 3 8B instance in FP16 consumes around 16GB of GPU memory, limiting how many concurrent requests you can serve. Quantization changes this equation dramatically. In this tutorial, I’ll walk you through quantizing Meta’s Llama 3 8B Instruct model to W8A8 (INT8 weights + INT8 activations) using LLMCompressor, then serving it with vLLM for production inference. Table of Contents What is W8A8 Quantization? Why Quantize? Prerequisites Project Setup The Quantization Recipe Quantization Implementation Serving with vLLM Testing Your Deployment Production Considerations Conclusion What is W8A8 Quantization? W8A8 stands for 8-bit Weights and 8-bit Activations. Instead of storing model parameters as 16-bit or 32-bit floating point numbers, we compress them to 8-bit integers. ...

The Popular Comparison A teenager learns to drive in 10 hours, but an AI system needs millions of simulations and millions of hours of simulated data. This comparison appears frequently in AI discussions, but once you look closely, it’s not fair. Here’s why. Where This Example Comes From I recently saw this example in an interview with Ilya Sutskever, co-founder of OpenAI who later started his own AI startup with significant investment backing. ...

When StarRocks introduced vector search, I had to test and see myself. Is it just a toy feature just bolted on an OLAP DB or not? Turns out this thing is actually good enough that you can fold your approximate nearest-neighbor (ANN) workloads into the same cluster that’s already powering your analytics dashboards. In this article, I decided to skip skip the benchmarking details and focus on what you actually need to get vector search running. ...

OpenAI recently released the GPT-5 family, introducing three distinct options for Pro users: GPT-5 Fast, GPT-5 Thinking, and GPT-5 Pro. While it’s common knowledge that GPT-5 Fast handles simple tasks and GPT-5 Pro tackles complex ones, the underlying mechanisms remain unclear to many users. This post provides a concise explanation of how each variant operates and when to use them effectively. GPT-5 Fast For the context of this article, we will use GPT-5 Fast as our base model, think of it as a black box optimized for speed, good old LLM. When you submit a query, it processes the request and delivers an answer quickly without extensive deliberation. ...

Table of Contents Introduction Prerequisites Project Setup Architecture Overview Exporting Models to ONNX Loading an ONNX Model Text Generation Pipeline Building the CLI Chat Interface Going Further Conversation Memory Temperature & Top-p Sampling Streaming Tokens Performance Optimizations Testing Deployment Considerations Conclusion TLDR ...