1326AB-B430E-21 基于Transformer結構的各類語言模型（Bert基于其encoder,Gpt-2基于其decoder）早已經在各類NLP任務上大放異彩，面對讓人眼花繚亂的transformer堆疊方式，你是否也會感到迷茫？沒關系，現在讓我們回到最初，再次看看transformer 本來的模樣——Rethinking the Value of Transformer Components。該文收錄已于COLING 2020。
眾所周知，一個完整的transformer結構可以切分成Encoder-self attention(“E:SA”)， Encoder-Feed Forward(“E:FF”)， Decoder-Self Attention(“D:SA”)， Decoder-Encoder Attention(“D:EA”) 和 Decoder-Feed Forward(“D:FF”) 5個sub-layer結構。文中作者采用了兩種度量方式確認這些sub-layer的重要程度。
方法稍后再談，先上干貨，實驗結果表明：
Decoder self-attention layers是最不重要的，而Decoder feed-forward layers是最重要的；
離模型的輸入和輸出越近的sub-layer要比其他的重要些；
在decoder里越靠后的encoder-attention layer要比之前的重要。
這些結果對不同的度量方法，數據集，初始化種子以及模型容量都能保持一致性。
▲Transformer結構圖
模塊重要性分析
所謂的重要性究竟是什么呢？論文認為，這主要包括兩個方面：
Contribution in information Flow，對于模型信息流的貢獻程度
Criticality in Representation Generalization，模型的模塊對參數的擾動表現出不同的魯棒性
Contribution in Information Flow
Transformer 最初是用來做機器翻譯任務的。所謂的information flow就是指數據如何從源語言經過Transformer的encoder和decoder最終成為目標語言的。如何衡量模型的每個部分對information flow做出的貢獻呢? 最直觀的想法就是去掉那個部分看看同樣條件下模型的效果如何。如果去掉那個部分，模型效果沒什么變化，那就說明該部分沒做什么貢獻，反之，如果刪掉該部分，模型效果顯著降低則說明它貢獻，沒它不行。作者采用了如下的量化方法：
公式中指的是去除第n個部分后模型整體的BLEU得分降。為了避免出現重要性指數出現負值和爆炸性下跌，作者將的值設定在[0,C]之間(真的會出現負重要性指數嗎？那樣倒挺好——模型變小，效果更好)。然后通過除以的得分降將的值進行了歸一化，這里作者設置的上限C值為基線模型的BLEU得分的1/10.
Criticality in Representation Generalization
這里說的criticality指的是模型的模塊對參數的擾動表現出不同的魯棒性。比方說，如果將某個模塊的參數重置為初始化參數，模型的表現變差，那么這個模塊就是critical的，否則就是non-critical的。有人在理論上將這個criticality給公式化了，而且他們表明這個criticality可以反映神經網絡的泛化能力。
作者便是參考了這個工作，對網絡的第n個模塊，定義
即初始權重和最終權重的一個凸組合。
那么第n個部分的criticality score就可以表示為
這個式子定量的說明了criticality是最小的能使模型在閾值的情況下保持性能。這個值越小說明該模塊越不重要，這里取的是 0.5 BLEU分。
兩種度量方法雖然都是基于模塊對模型表現的影響的，但是又有不同之處。Contribution score可以看成是 hard metric(完全刪除模塊)，而 Criticality score可以看成是一種soft metric,它衡量的是在保證模型表現的前提下模塊參數能多大程度的回卷。
1326AB-B430E-21 AMAT APPLIED MATERIALS 0050-25830 VACUUM FITTING
Applied Micromechanics of Porous Materials NEW
Grand Eagle - Great books, Great prices, Great service.
Applied Materials 0020-82550 HOLDER (MRS) P19500
APPLIED MATERIALS P36W-17 PRESSURE SWITCH 10 PSIG
Applied Electromagnetism and Materials NEW
Applied Strength of Materials NEW by Robert L. Mott
Grand Eagle - Great books, Great prices, Great service.
Applied Statics and Strength of Materials NEW
Grand Eagle - Great books, Great prices, Great service.
Applied Materials AMAT Circuit Breaker, 0680-00608
APPLIED MATERIALS E36W-74 VACUUM SWITCH 830 TORR
Applied Materials AMAT Kalrez O-Ring, 3700-01807
APPLIED MATERIALS INC. HOLDER (MRS) P19500
Applied Strength of Materials by Robert L. Mott (200...
Fast Shipping ~ Quick Processing ~ BRAND NEW !!!!
One-day shipping available
APPLIED MATERIALS 0100-09127 PCB ASSY - TESTED
APPLIED MATERIALS HEATER BAND FOR TMS KIT, 1410-01124
APPLIED MATERIALS SHOWER HEAD,5B PUTTER ETCH 0020-09933
APPLIED MATERIALS 0100-76053 PCB ROBOT INTERCONNECT
applied materials 8300 brake 0140-00024
NEW - APPLIED MATERIALS DIO LOADING BOARD 200113D01
Applied Materials 0050-33586 TBG Connector AMAT
Applied Materials 0200-09657 Endpoint Window AMAT
Applied Materials 0200-20161 Insulator, Lamp AMAT
Applied Materials 3400-01134 Flex Hose AMAT
Applied Materials MKS LDM B12PA2CC1 AMAT
Applied Materials 5000 Cleanroom 8” Robot
AMAT APPLIED MATERIALS 3870-01306 VALVE PNEUMATIC NEW
APPLIED MATERIALS COVERING PIPE, 0020-0930
Applied Materials 0200-20059 INSULATOR 5" SMRMF PCII
Applied Materials 0020-81482
Applied Materials 0020-82822
PRINTED CIRCUIT BOARD APPLIED MATERIALS 0100-09106- NEW
APPLIED MATERIALS SHIELD 200MM PCIIE-SLT CHAMBERS C & D
APPLIED MATERIALS LOWER SHIELD 6" WAFER ,13" SOURCE
AMAT Applied Materials 0050-RB314
NEW APPLIED MATERIALS AMAT 3870-02260 VALVE ADAPTOR
APPLIED MATERIALS PRI REV D KX00002 BM70020 CPU BOARD
APPLIED MATERIALS AMAT COVER RING 6" PN 0020-23181
AMAT Applied Materials Heater 0090-00798
NEW APPLIED MATERIALS CABLE PORT SERVER 0140-77615
APPLIED MATERIALS ELECTROSTATIC CHUCK PART # 0040-31815
Applied Materials # 0190-02046 24 Port Valve Manifold
Applied Materials AMAT Photo-micro Sensor, 1400-01329
APPLIED MATERIALS 1410-01098 AMAT HEATER JKT FORELINE
One-day shipping available
Applied Materials AMAT Chemraz O-Ring, 3700-02181
APPLIED MATERIALS 0090-SCNSR KTC SENSOR ASSY AMAT
APPLIED MATERIALS 300mm GAS INTLK 0100-00580 WARRANTY!!
APPLIED MATERIALS MAIN RAME INTERLOCK p/n:0190-24338
Applied Materials 8100 D System DC Power Supplies AMAT
APPLIED MATERIALS BEARING P/N: 0190-77184 - NEW
Applied Materials AMAT Endpoint Window, 0200-35292
Applied Materials AMAT Accusync, 0246-00040
Applied Materials AMAT K1S Weldment, 0050-88784
Applied Materials AMAT VCR Elbow Weldment, 0050-84245
Applied Materials AMAT VCR K1S Weldment, 0050-52717
Applied Materials AMAT VCR K1S Weldment, 0050-52719
Applied Materials AMAT VCR Weldment, 0050-45870
Applied Materials AMAT Valve Regulator Assy, 0190-24421
Applied Materials AMAT Kalrez O-ring, 3700-00588
Applied Materials AMAT Vacuum Sensor, 1400-01298
APPLIED MATERIALS 0020-02680 GUIDE RETAINER, LIFT PIN
Applied Materials AMAT Kalrez O-Ring, 3700-04039
Applied Materials 0020-04076
Materials Science and Applied Physics: Second Mexican M
Grand Eagle - Great books, Great prices, Great service.
APPLIED MATERIALS INSERT PINS, SILICON,125 MM,0200-3671
APPLIED MATERIALS CARRIER, COMPLIANT, 0020-31463
Applied Materials 0020-25662 SHIELD, CLAMP, SCRWLES- CO
NEW Applied Materials/AMAT Shield Bowl PN: 0020-27160
NEW Applied Materials/AMAT Shield Bowl PN: 0020-29347
AMAT Applied Materials Bracket 0020-13999
APPLIED MATERIALS 0140-00545 COVER SWITCH & OVERTEMP
APPLIED MATERIALS 0620-01061 MTR pump Cable 10 FT
APPLIED MATERIALS 3400-01066 NEW
APPLIED MATERIALS DIP294 DEVICE NET I/O BLOCK
Applied Materials AMAT Kalrez O-Ring, 3700-03771
Applied Materials AMAT RF Connector, 0720-03402
APPLIED MATERIALS 0100-00370 BST LEVEL SENSOR BD ASSY
Applied Materials AMAT Bearing, 3060-00147
APPLIED MATERIALS 0140-09014 HARNESS ASSY,CONTROLLER SY
APPLIED MATERIALS 0090-16024, PA 12100R6517 SENSOR
APPLIED MATERIALS 0200-09347A, AMAT COLLAR 150MM FLAT
Applied Materials AMAT Chemraz O-Ring, 3700-02580
APPLIED MATERIALS 0190-00211 WASCO VACUUM SWITCH
Applied Materials AMAT VCR Weldment, 0050-01041
Applied Materials AMAT Surface Mount Valve, 3870-00538
APPLIED MATERIALS 0150-76035 WASCO SWITCH 2 PSIG decr
APPLIED MATERIALS RING, GRAPHITE, 125mm 0200-09092
Applied Materials AMAT 10 Pack Lamp 0190-14083
APPLIED MATERIALS 1720-01623 E36W-H35 VACUUM SWITCH
Applied Materials AMAT KALREZ O-RING, 3700-01929
APPLIED MATERIALS INC. KIT, CLIP P19500-AMI (NVNG CL
APPLIED MATERIALS HEATER THROTTLE VALVE 1410-01345
LOT APPLIED MATERIALS FUJIKIN MANIFOLD VALVE DIAPHRAGM
NEW Materials Science and Applied Physics: Second Mexic
Expedited shipping available
Applied Materials 2 Position DPS Data Switch 1270-01686
ELECTRODE SUPPRESSION - APPLIED MATERIALS - 0020-86280
Enlarge
APPLIED MATERIALS 9150-04369 NEW
Applied Materials AMAT VCR Weldment, 0050-61611
Applied Materials AMAT IR Halogen Lamp, 1010-00237
APPLIED MATERIALS 12-8892-010 PITTMAN GM9413J820 19.7:1
Applied Materials Capture Ring 145MM 0200-35350 AMAT
Applied Materials AMAT Pressure Gauge, 0190-14247
Applied Materials AMAT Sensor Head, 1400-00073
|
在線咨詢 
