 -- Gmail Smart Compose)
Introduction说白了这一章就是教我们LLM补全的设计架构Gmails Smart Compose feature [1] assists users by suggesting the next few words asthey write an email.Clarifying Requirementsseems 我们作为被面试人需要给面试官提一串需求来去定位我们的设计function- suggest range? personalized?dataset - size? range? language?output -biased?system - user number? real-time?Specifying the systems input and outputChoosing a suitable ML approachrnn vs transformerVarious techniques are introduced to reduce the complexity of attention.Group Attention [6] and FlashAttention [7]Data Preparationdata source: general data/email dataText cleaningRemove non-English text/ confidential information(脱敏/ irrelevant characters or symbols/ duplicated dataText normalization时间日期电话等归一化Text tokenizationTokenization level目前最好用的是subword-levelTokenizer chooseByte-Pair Encoding (BPE)SentencePiece .Tiktokentoken indexingconverting textual tokens into integer numbersModel DevelopmentTransformer Architecture这个我们这里也不展开了平时算法笔记也做的比较详细了Encoder-only-- understanding the overallmeaning of a text.Decoder-only-- processes the input sequence and generates a newsequence iterativelyEncoder-decoder-- encoder component processes the input sequence and a decoder uses that processed information to generate the output sequence e.g 翻译)在这个续写任务中我们当然就用decoder啦A decoder-only Transformer consists of the following components:Text embedding把每个 token ID 映射成高维向量作为神经网络真正处理的输入。Positional encodingpositional encoding provides the Transformer with position information for each token in the input sequence 一般情况下选 fixed 准没错Fixed positional encodingpros: Efficiency, Support for long sequencescons: Predefined limits(就是字数不能超), Suboptimal performanceLearned positional encodingprosOptimal performancecons: Inefficiency Lack of generalizationTransformerTransformer architecture consists of a stack of blocks. Each block contains theMulti-head attention:This layer updates each embedding by using the attention mechanism. The attention mechanism captures the relationships in the sequence by allowing each embedding to attend to its preceding embeddings. Due to the nature of its mechanism, multi-head attention is commonly known as self-attention, a term well use throughout the rest of this book.Feed forward:This layer applies two linear transformations, with a ReLU activation in between, to each embedding in the sequence independently.Prediction headthe final component in a decoder-only Transformer, translates the Transformers output into probabilities for every token in the vocabulary.These probabilities are used to choose the most likely next token.Trainingtwo-stage trainingwhy two stage?Adaptability, Improved generalization, Fast finetuning, Handling data scarcity, Mitigating overfitting, Resource optimizationpretraining一般工程中我们都拿这个阶段已经训练好的模型The purpose of pretraining is to develop a model capable of understanding natural language, including syntax, common knowledge, and language structures.ML objective and loss functionIn the case of text generation, the most commonly used ML objective is next-token prediction.btw 这里注意区分一下masked prediction训练方式典型模型输入能看什么训练目标更适合Next-token predictionGPT / LLaMA / Qwen 这类生成式 LLM只能看前文预测下一个 token文本生成、对话、续写Masked prediction / MLMBERT可以同时看左右文预测被[MASK]掉的词文本理解、分类、检索、匹配Cross-entropy loss这个也不多解释了就是奖励应该出现的那个tokenIn practice, the model processes all token lengths within a sequence in parallel. This allows it to compute theloss for each token position simultaneously. Parallelizing this step speeds up training by handling multiple tokens at once, instead of sequentially.Transformer 会同时为每个位置输出一个 next-token 概率分布 然后每个位置都和正确答案算一次 cross-entropy loss最后加起来或取平均causal mask 防止模型在预测下一个 token 时偷看后面的 token。* BOS(beginning of sequence) 和 EOS(end of sequence) 是 序列起始符/结束符号finetuning这里就用到专项(task- specified)数据集了邮件)ML objective and loss function同 pretrain next-token prediction cross-entropy loss缺失信息处理这个比较工程比如你打了一个 dear 模型怎么知道要写谁的名字呢于是他去更深的信息去翻Combining various inputs总之放到工程就可以理解为把各种信息拼到一块吧比如一个json 可以包含各个类别的信息Samplingprocess of using a trained generative model to generate new dataThe process continues untill the model predicts the EOS token一般情况下选 deterministicConsistency, Better handling of common phrases, Reduced risk of inappropriate suggestionsdeterministicgenerate text in a deterministic way, that is, without randomness or variability in the output.At each step of token generation, the modelselects the token with the highest probabilityfrom the predicted distribution.prosConsistency, Predictable outputscons: Lack of diversity, Repetitive text我草原来这种重复的罪魁祸首是greedy search哇Greedy searchalways selects the token with the highest probability as the next tokenBeam search保留多条高分路径track multiple potential sequences of tokens simultaneously. At each step, the model calculates the probabilities for the next possible tokens for each sequence and selects the top-k most probable sequences.Initialization:Start with the users partial email as the input to the trained model. The model predicts the probability distribution for the next token. Beam search selects thetop three tokens with the highest probabilities.Expansion:For each top three sequence, pass it to the model and obtain the probabilities of the next token.Pruning:Select thetop three sequences based on their cumulative probabilities.Once the beam search algorithm has stopped, we select the sequence with thehighest cumulative probabilityas the output.consLimited diversity Struggle with long sequencesStochastic 随机的Stochastic sampling methods introduce randomness into the generation process.For example, at each step of token generation, the model samples from the predicted distribution based on the probabilities assigned to each token.prosDiversity NoveltyconsInconsistency Unexpected outputs注意: stochastic sampling 不是不会重复而是相比 greedy / beam 这种 deterministic decoding它更不容易被固定的最高概率路径锁死。在邮件补全的场景下补全的内容通常比较短也不期望多样性没有必要上beam searchEvaluationOffline evaluation metricsensure the models performance is acceptable before deploying it to productionOffline evaluation uses pre-collected and historical data to evaluate a models performance除了 PerplexityEcact match 还有 BLEU score and ROUGE-N 后边讲Perplexity 困惑度This metric measures howaccurately the model predicts the exact sequence of tokens present in text data.越低越好ExactMatchNpercentage of generated phrases that are exactly N words long and that match the first N words of the ground-truth text.Online evaluation metricshow a model performs in real time as users interact with the systemdefined based on specific requirements and needsUser engagement metricsAcceptance rateUsage rateEffectiveness metricsAverage completion timeLatency metricsSystem response timeQuality metricsFeedback rateHuman evaluationML System Design这个好啊有一说一感觉平时工作都是在前面的部分针对Smart Compose整个流程主要是Monitoring:The triggering service monitors the users activity as they type.Triggerring:The service triggers the phrase generator once it identifies specific patterns.Beam search:The phrase generator employs beam search to get top-k potential completions from the trained model.Filtering:The phrase generator interacts with the filtering component to remove long suggestions and those with low confidence scores.Post-processing:The completion with the highest score is picked and passed to the post-processing service. The service replaces gender-specific pronouns and adjusts sensitive terms.Display suggestion:The suggestion is displayed to the user for their consideration.Triggering servicetriggering service activates the Smart Compose feature by monitoring user activity such as keystrokes.判断需要的时候再触发Phrase generator这个就是我们刚刚讨论了老半天的算法部分It generates the most likely completion based on the partial text the user has already typed.Removing long suggestionsRemoving low-confidence suggestions好简单粗暴....Post-processing serviceaddresses potential biases before suggestions are presented to the userPronoun replacementGender-neutral word replacementLexical ( 词汇) analysis for sensitive termsNSFW (Not Safe For Work) content filteringOther Question to prepare下回再说现在主要是给书过一遍Supporting Smart Compose in multiple languages [28].Personalizing suggestions [28].Incorporating additional context for better predictions [28].Understanding how different tokenization algorithms work, such as BPE [11], SentencePiece [12], and WordPiece [29].Understanding different ML objectives such as masked language modeling (MLM) and its variations [18].The multi-token prediction objective and its pros and cons [30].Balancing quality and inference latency [28].
