def wals_roberta(sentences, model, tokenizer, pca_components, alpha=1e-4): emb = encode(sentences) # (n, d) # Whiten by inverse singular values U, S, Vt = torch.pca_lowrank(emb, q=pca_components) S_inv = 1.0 / torch.sqrt(S**2 + alpha) W = Vt.T @ torch.diag(S_inv) @ Vt # projection matrix return emb @ W
Allows a model trained in English to apply "structural logic" to a low-resource language it hasn't seen much of before. Zero-Shot Learning wals roberta sets upd
By informing a RoBERTa model about the grammatical structure (e.g., word order) of a target language via WALS data, the model can perform better on that language even if it has never seen it during training. alpha=1e-4): emb = encode(sentences) # (n
Elimination of overlapping parameters that previously caused system conflicts. Vt = torch.pca_lowrank(emb