Lecture 2 Autoregressive Models

pdf

School

University of Cincinnati, Main Campus *

*We aren’t endorsed by this school

Course

OPTIMIZATI

Subject

Computer Science

Date

Oct 30, 2023

Type

pdf

Pages

113

Uploaded by BrigadierGorillaMaster2190

Report
±S294-158 ²eep Unsupervised !Learning 00-ieter ±IIFbIIFbeel, ==:Xi (00-eter) ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan 663U² ´erkeley !Lecture 2 !Likelihood Models: ³utoregressive Models
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - &&odern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoregressive &&odels - 330ReJJGcurrent ''$eural ''$ets - &&asking-IIFbased &&odels 2
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - Motivation - 441Simple generative models: histograms - &&odern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoregressive &&odels - 330ReJJGcurrent ''$eural ''$ets - &&asking-IIFbased &&odels 3
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels !%%ikelihood-IIFbased models Problems we’d like to solve: - ¶enerating data: synthesizing images, videos, speeJJGch, text - ²ompressing data: JJGconstruJJGcting effiJJGcient JJGcodes - ±nomaly deteJJGction !Likelihood-based models: estimate p data from samples x (1) , ș , x (n) ~ p data (x) !%%earns a distriIIFbution p that allows: - ²omputing p(x) for arIIFbitrary x - 441Sampling x ~ p(x) 552Today: discrete data 4
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels µesiderata <<9We want to estimate distriIIFbutions of complex, high-dimensional data - ± 128x128x3 image lies in a ~50,000-dimensional spaJJGce <<9We also want JJGcomputational and statistiJJGcal effiJJGcienJJGcy - ·ffiJJGcient training and model representation - ·xpressiveness and generalization - 441Sampling quality and speed - ²ompression rate and speed 5
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - Simple generative models: histograms - &&odern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoregressive &&odels - 330ReJJGcurrent ''$eural ''$ets - &&asking-IIFbased &&odels 6
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels !%%earning: ·stimate frequenJJGcies IIFby JJGcounting 330ReJJGcall: the goal is to estimate p data from samples x (1) , ș , x (n) ~ p data (x) 441Suppose the samples take on values in a finite set {1, … , k} 552The model: a histogram - (330Redundantly) desJJGcriIIFbed IIFby k nonnegative numIIFbers: p 1 , … , p k - 552To train this model: JJGcount frequenJJGcies p i = (# times i appears in the dataset) / (# points in the dataset) 7
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¸nferenJJGce and 441Sampling °nference (querying p i for arIIFbitrary i): simply a lookup into the array p 1 , … , p k Sampling (lookup into the inverse JJGcumulative distriIIFbution funJJGction) 1. ¹rom the model proIIFbaIIFbilities p 1 , … , p k , JJGcompute the JJGcumulative distriIIFbution ¹ i = p 1 + 䐃 + p i for all i 䌜 {1, … , k} 2. µraw a uniform random numIIFber u ~ [0, 1] 3. 330Return the smallest i suJJGch that u ≤ ¹ i ³re we done? 8
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¹ailure in high dimensions ''$o, IIFbeJJGcause of the curse of dimensionality . ²ounting fails when there are too many IIFbins. - (´inary) &&''$¸441S552T: 28x28 images, eaJJGch pixel in {0, 1} - 552There are 2 784 ≈ 10 236 proIIFbaIIFbilities to estimate - ±ny reasonaIIFble training set JJGcovers only a tiny fraJJGction of this - ·aJJGch image influenJJGces only one parameter. ''$o generalization whatsoever! 9
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-roIIFblematiJJGc even for single variaIIFble 10 learned histogram = training data distriIIFbution → often poor generalization
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-arameterized µistriIIFbutions 11 ¹itting a parameterized distriIIFbution often generalizes IIFbetter
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441Status - °ssues with histograms - ³igh dimensions: won’t work - ·ven 1-d: if many values in the domain, prone to overfitting - Solution: function approximation . ¸nstead of storing eaJJGch proIIFbaIIFbility, store a parameterized funJJGction p θ (x) 12
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - Modern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoregressive &&odels - 330ReJJGcurrent ''$eural ''$ets - &&asking-IIFbased &&odels 13
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - Modern neural autoregressive models - ->Er>EmeterZized dZistrZiFMbutZions >End m>ExZimum lZikelZiYhood - ±utoregressive &&odels - 330ReJJGcurrent ''$eural ''$ets - &&asking-IIFbased &&odels 14
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels !%%ikelihood-IIFbased generative models 330ReJJGcall: the goal is to estimate p data from x (1) , ș , x (n) ~ p data (x) ''$ow we introduJJGce function approximation : learn θ so that p θ (x) ≈ p data (x). - ³ow do we design funJJGction approximators to effeJJGctively represent JJGcomplex joint distriIIFbutions over x, yet remain easy to train? - 552There will IIFbe many JJGchoiJJGces for model design, eaJJGch with different tradeoffs and different JJGcompatiIIFbility JJGcriteria. ²esigning the model and the training procedure go hand-in-hand. 15
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¹itting distriIIFbutions - ¶iven data x (1) , ș , x (n) sampled from a “true” distriIIFbution p data - 441Set up a model JJGclass: a set of parameterized distriIIFbutions p θ - 00-ose a searJJGch proIIFblem over parameters - <<9Want the loss funJJGction + searJJGch proJJGcedure to: - <<9Work with large datasets (n is large, say millions of training examples) - >>;Yield θ suJJGch that p θ matJJGches p data — i.e. the training algorithm wdnr_os. 552Think of the loss as a distanJJGce IIFbetween distriIIFbutions. - ''$ote that the training proJJGcedure JJGcan only see the empiriJJGcal data distriIIFbution, not the true data distriIIFbution: we want the model to generalize. 16
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&aximum likelihood - &&aximum likelihood: given a dataset x (1) , ș , x (n) , find θ IIFby solving the optimization proIIFblem - 441StatistiJJGcs tells us that if the model family is expressive enough and if enough data is given, then solving the maximum likelihood proIIFblem will yield parameters that generate the data - ·quivalent to minimizing $$!%% divergenJJGce IIFbetween the empiriJJGcal data distriIIFbution and the model 17
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441StoJJGchastiJJGc gradient desJJGcent &&aximum likelihood is an optimization proIIFblem. ³ow do we solve it? Stochastic gradient descent (441S¶µ). - 441S¶µ minimizes expeJJGctations: for f a differentiaIIFble funJJGction of θ , it solves - <<9With maximum likelihood, the optimization proIIFblem is - Why maximum likelihood + S´²? ¸t works with large datasets and is JJGcompatiIIFble with neural networks. 18
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels µesigning the model - ey requirement for maximum likelihood + S´²: efficiently compute log p(x) and its gradient - We will choose models p · to be deep neural networks , whiJJGch work in the regime of high expressiveness and effiJJGcient JJGcomputation (assuming speJJGcialized hardware) - µow exactly do we design these networks? - ±ny setting of θ must define a valid proIIFbaIIFbility distriIIFbution over x: - log p θ (x) should IIFbe easy to evaluate and differentiate with respeJJGct to θ - 552This JJGcan IIFbe triJJGcky to set up! 19
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ´ayes nets and neural nets Main idea : plaJJGce a ¶ayes net struJJGcture (a direJJGcted aJJGcyJJGcliJJGc graph) over the variaIIFbles in the data, and model the JJGconditional distriIIFbutions with neural networks. Reduces the problem to designing conditional likelihood-based models for single variables . <<9We know how to do this: the neural net takes variaIIFbles IIFbeing JJGconditioned on as input, and outputs the distriIIFbution for the variaIIFble IIFbeing prediJJGcted. 20
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - Modern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoreXgressZive #odels - 330ReJJGcurrent ''$eural ''$ets - &&asking-IIFbased &&odels 21
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ±utoregressive models - ¹irst, given a ´ayes net struJJGcture, setting the JJGconditional distriIIFbutions to neural networks will yield a traJJGctaIIFble log likelihood and gradient. ¶reat for maximum likelihood training! - ´ut is it expressive enough? >>;Yes, assuming a fully expressive ´ayes net struJJGcture: any joint distriIIFbution JJGcan IIFbe written as a produJJGct of JJGconditionals - 552This is JJGcalled an autoregressive model . 441So, an expressive ´ayes net struJJGcture with neural network JJGconditional distriIIFbutions yields an expressive model for p(x) with traJJGctaIIFble maximum likelihood training. 22
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ± toy autoregressive model 552Two variaIIFbles: x 1 , x 2 &&odel: p(x 1 , x 2 ) = p(x 1 ) p(x 2 |x 1 ) - p(x 1 ) is a histogram - p(x 2 |x 1 ) is a multilayer perJJGceptron - ¸nput is x 1 - ))&utput is a distriIIFbution over x 2 (logits, followed IIFby softmax) 23
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&ne funJJGction approximator per JJGconditional µoes this extend to high dimensions? - 441Somewhat. ¹or d-dimensional data, ))&(d) parameters - &&uJJGch IIFbetter than ))&(exp(d)) in taIIFbular JJGcase - <<9What aIIFbout text generation where d JJGcan IIFbe arIIFbitrarily large? - !%%imited generalization - ''$o information sharing among different JJGconditionals 441Solution: share parameters among JJGconditional distriIIFbutions. 552Two approaJJGches: - 330ReJJGcurrent neural networks - &&asking 24
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - Modern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoreXgressZive #odels - 07eNcurrent $eur>El $ets - &&asking-IIFbased &&odels 25
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 330R''$''$ autoregressive models - JJGchar-rnn [Karpathy, 2015] Sequence of characters Character at ONO th position 26
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&''$¸441S552T 27 ■ ³andwritten digits ■ 28x28 ■ 60,000 train ■ 10,000 test ■ ))&riginal: greysJJGcale ■ “´inarized &&''$¸441S552T” -- 0/1 (IIFblaJJGck/white)
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 330R''$''$ on &&''$¸441S552T 28
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 330R''$''$ with 00-ixel !%%oJJGcation ±ppended on &&''$¸441S552T 29 ■ ±ppend (x,y) JJGcoordinates of pixel in the image as input to 330R''$''$
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - Modern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoreXgressZive #odels - 330ReJJGcurrent ''$eural ''$ets - #>EskZinXg-FMb>Esed #odels - #±°² - &&asked ²onvolutions - <<9Wavenet - 00-ixel²''$''$ (+variations) 30
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asking-IIFbased autoregressive models 441SeJJGcond major IIFbranJJGch of neural ±330R models ■ ey property: parallelized JJGcomputation of all JJGconditionals ■ &&asked &&!%%00- (&&±µ·) ■ &&asked JJGconvolutions & self-attention ■ ±lso share parameters aJJGcross time 31
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±utoenJJGcoder for µistriIIFbution ·stimation (&&±µ·) 32
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±utoenJJGcoder for µistriIIFbution ·stimation (&&±µ·) 33 ´eneral principle
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&±µ· on &&''$¸441S552T 34
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±utoenJJGcoder for µistriIIFbution ·stimation (&&±µ·) ± TFLEVFLEQ² RSVQFLEP ,) [JPITMLRKSLX ± \² PFLE]JPIV TMRTYX ± ]² FLEYXSVJPILRKVJPIWWTMZJPI FLEHNGXTMZFLEXTMSRW QFLEWO !# LRKJPIXDJCPTMRJPIFLEVDJCFLEVDJCQFLEWO ³ TMRDJCWTM^JPI ´ SYXDJCWTM^JPI µ ± HNGVJPIFLEXJPI QFLEWO SKQJ TFLEXXJPIVR ± FLEVVFLE]³A@FA@F¶·´ ¸·´ ¸·BHA´ ± A@F¶·´ ¶·´ ¸·BHA´ ± A@F¶·´ ¶·´ ¶·BHABHA´ IOHX]TJPI!#KQJPSFLEX¹ºµ ] !# XKQJ · QFLEXQYP ³ \ ´ TFLEVFLEQ » QFLEWO µ 35
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&±µ· results 36
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&±µ· results 37
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&±µ· -- µifferent ))&rderings 38 330Random 00-ermutation ·ven then ))&dd ¸ndiJJGces 330Rows (330Raster 441SJJGcan) ²olumns 552Top to &&iddle, ´ottom to &&iddle ±ll orderings aJJGchieve roughly the same IIFbits per dim, IIFbut samples are different
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&±µ·: &&ultiple ))&rderings 39
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - Modern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoreXgressZive #odels - 330ReJJGcurrent ''$eural ''$ets - #>EskZinXg-FMb>Esed #odels - &&±µ· - #>Esked ³onvolutZions - 9@>Evenet - 00-ixel²''$''$ (+variations) 40
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked 552Temporal (1µ) ²onvolution ● Easy to implement, masking part of the conv kernel ● Constant parameter count for variable-length distribution! ● Efficient to compute, convolution has hyper-optimized implementations on all hardware However ● Limited receptive field, linear in number of layers x i p(x i+1 | x <=i ) 41
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels <<9Wave''$et ● Improved receptive field: dilated convolution, with exponential dilation ● Better expressivity: Gated Residual blocks, Skip connections 42
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels <<9Wave''$et on &&''$¸441S552T 43
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels <<9Wave''$et with 00-ixel !%%oJJGcation ±ppended on &&''$¸441S552T 44 ■ ±ppend (x,y) JJGcoordinates of pixel in the image as input to <<9Wave''$et
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked 552Temporal (1µ) ²onvolution ± 328SVJPI JPIKQJKQJTMHNGTMJPIRX TMQTPJPIQJPIRXFLEXTMSR TSWWTMGMFPJPI GMF] ± TFLEIOHIOHTMRLRK TMRWXJPIFLEIOH SKQJ QFLEWOTMRLRK OJPIVRJPIPW ± O² WTM^JPI SKQJ OJPIVRJPIP ± OJPIVRJPIP² HNGSRZSPYXTMSR [JPITMLRKSLXW TFLEIOHIOHJPIIOHDJC\ !# XKQJ · TFLEIOH ³ \ ´ A@F ³ ´ µ´ ³ O ¼ ¸ ´ µ´ ³ ´ µ´ ³ ´ µ BHAµ ] !# XKQJ · RR · HNGSRZºIOH ³ TFLEIOHIOHJPIIOHDJC\ ´ OJPIVRJPIP ´ TFLEIOHIOHTMRLRK !# ½<;A'21/.*½ µ 45
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ))&utline - &&otivation - 441Simple generative models: histograms - Modern neural autoregressive models - 00-arameterized distriIIFbutions and maximum likelihood - ±utoreXgressZive #odels - 330ReJJGcurrent ''$eural ''$ets - #>EskZinXg-FMb>Esed #odels - &&±µ· - #>Esked ³onvolutZions - <<9Wavenet - -Zixel³$$ (+v>ErZi>EtZions) 46
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked 441Spatial (2µ) ²onvolution - 00-ixel²''$''$ ■ ¸mages JJGcan IIFbe flatten into 1µ veJJGctors, IIFbut they are fundamentally 2µ ■ <<9We JJGcan use a masked variant of ²onv''$et to exploit this knowledge ■ ¹irst, we impose an autoregressive ordering on 2µ images: This is called raster scan ordering. (Different orderings are possible, more on this later) 47
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-ixel²''$''$ ■ µesign question: how to design a masking method to oIIFbey that ordering? ■ ))&ne possiIIFbility: 00-ixel²''$''$ (2016) 48
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
:<;`VNW[Z^THJIR_^ :<;HJIR^TWV]SQZP_UOX 0 255 :<;]SQZPMLUMV KJS`_ 2HJIRYX`V_U ]\HJIR_U MLUMV_U 687`VYXMLU
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-ixel²''$''$ ■ 00-ixel²''$''$-style masking has one proIIFblem: IIFblind spot in reJJGceptive field 72
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¶ated 00-ixel²''$''$ ■ ¶ated 00-ixel²''$''$ (2016) introduJJGced a fix IIFby JJGcomIIFbining two streams of JJGconvolutions This is easy, we know how to do 1D masked conv How? 73
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¶ated 00-ixel²''$''$ ■ ;;8VertiJJGcal staJJGck: through padding, aJJGctivations at X th row only depend on input IIFbefore X th row 74
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¶ated 00-ixel²''$''$ ■ ¸mproved ²onv''$et arJJGchiteJJGcture: ¶ated 330Res''$et ´loJJGck 75
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¶ated 00-ixel²''$''$ ■ ´etter reJJGceptive field + more expressive arJJGchiteJJGcture = IIFbetter performanJJGce 76
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-ixel²''$''$++ ■ &&oving away from softmax: we know nearIIFby pixel values are likely to JJGco-oJJGcJJGcur! 77
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 330ReJJGcap: !%%ogistiJJGc distriIIFbution pdf cdf = sigmoid((x - mu) / scale) 78
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&ixture of !%%ogistiJJGcs -- µisJJGcrete µistriIIFbution 79
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ·x. 552Training &&ixture of !%%ogistiJJGcs 80
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-ixel²''$''$++ ■ ²apture long dependenJJGcies effiJJGciently IIFby downsampling 81
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-ixel²''$''$++ 82
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±ttention ■ ± reJJGcurring proIIFblem for JJGconvolution: limited reJJGceptive field -> hard to JJGcapture long-range dependenJJGcies ■ (441Self-)±ttention: an alternative that has ■ unlimited reJJGceptive field!! ■ also ))&(1) parameter sJJGcaling w.r.t. data dimension ■ parallelized JJGcomputation (versus 330R''$''$) 83
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ±ttention Self-attention when q i also generated from x 84
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441Self-±ttention Convolution Self-attention 85
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±ttention - masked(k i , q) * 10 10 - masked(k i , q) * 10 10 86
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±ttention ■ &&uJJGch more flexiIIFble than masked JJGconvolution. <<9We JJGcan design any autoregressive ordering we want ■ ±n example: Zigzag ordering - How to implement with masked conv? - Trivial to do with masked attention! 87
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±ttention + ²onvolution 88
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±ttention + ²onvolution 89 ¶ated 00-ixel²''$''$ 00-ixel²''$''$++ 00-ixel441S''$±¸!%%
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&ulti-³ead 441Self-±ttention on &&''$¸441S552T 90
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked ±ttention + ²onvolution 91
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441Sample 22/Quality 92 <<9WhiJJGch set of samples are generated IIFby a ¶±''$ versus an ±330R model?
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ±330R models JJGcan have good samples 93 ■ ¶ood samples JJGcan IIFbe aJJGchieved IIFby seleJJGctive IIFbits JJGconditioning ■ ¶raysJJGcale 00-ixel²''$''$ ■ 441SuIIFbsJJGcale 00-ixel ''$etwork
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ²lass-²onditional 00-ixel²''$''$ 94 µow to condition? 0257*+ 687_UMV°PY`V[Z MV_ULKT`VMLUQZP_UOX `VNW [ZPYMV ]SHJIRKJSMV]SZY ;=<E9/6,57*+ ^T\[]S[ZQZPWV]S`_QZP_UOX KJS`_ MLUQZPNWNWMVYXMV_U[Z ]SMVHJIRYX_UMVMLU ^]MVQZPOXPY[Z ^THJIR[ZYXQZPLKTMVZY QZP_U MVHJIRLKTPY LKT`V_U]\`V]S\[[ZQZP`V_UHJIR]S ]SHJIR`_MVYX± HJIR_UMLU HJIRMLUMLUMVMLU HJIRZY HJIR KJSQZPHJIRZY LKTPYHJIR_U_UMV]S°^]QZPZYMV HJIR_UMLU KJSYX`VHJIRMLULKTHJIRZY[ZMVMLU ZYWVHJIR[ZQZPHJIR]S]S`_
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ³ierarJJGchiJJGcal ±utoregressive &&odels with ±uxiliary µeJJGcoders 95 De Fauw, Jeffrey, Sander Dieleman, and Karen Simonyan. "Hierarchical autoregressive image models with auxiliary decoders." HGFWXW?>ONO[\[\ VUVUWXWKJVUVUWXWONOTSTYZY HGFWXW?>ONO[\[\*$!) #° $)## (2019). APA
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ¸mage 441Super-330Resolution with 00-ixel²''$''$ ■ ± 00-ixel²''$''$ is JJGconditioned on 7 x 7 suIIFbsampled &&''$¸441S552T images to generated the JJGcorresponding 28 x 28 image 96
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-ixel 330ReJJGcursive 441Super 330Resolution 97
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ³ierarJJGchy: ¶raysJJGcale 00-ixel²''$''$ 98 ■ µesign an autoregressive model arJJGchiteJJGcture that takes advantage of the struJJGcture of data ■ !%%earn a 00-ixel²''$''$ on IIFbinary images, and a 00-ixel²''$''$ JJGconditioned on IIFbinary images to generate JJGcolored images
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 00-ixel²''$''$ &&odels with ±uxiliary ;;8VariaIIFbles for ''$atural ¸mage &&odeling 99
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ''$eural autoregressive models: the good ´est in JJGclass modelling performanJJGce: ■ expressivity - autoregressive faJJGctorization is general ■ generalization - meaningful parameter sharing has good induJJGctive IIFbias -> 441State of the art models on multiple datasets, modalities 100
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&asked autoregressive models: the IIFbad ● Sampling each pixel = 1 forward pass! ● 11 minutes to generate 16 32-by-32 images on a Tesla K40 GPU 101
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441Speedup IIFby JJGcaJJGching aJJGctivations 102 https://github.com/PrajitR/fast-pixel-cnn
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441Speedup IIFby JJGcaJJGching aJJGctivations 103 https://github.com/PrajitR/fast-pixel-cnn
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441Speedup IIFby IIFbreaking autoregressive pattern ■ ))&(d) -> ))&(log(d)) IIFby parallelizing within groups {2, 3, 4} ■ ²annot JJGcapture dependenJJGcies within eaJJGch group: this is a fine assumption if all pixels in one group are JJGconditionally independent ■ &&ost often they are not, then you trade expressivity for sampling speed 104
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels &&ultisJJGcale 00-ixel²''$''$ Improved sampling speed More limited modelling capacity 105
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441SJJGcaling ±utoregressive ;;8Video &&odels 106 [ Dirk Weissenborn, Oscar Tackstrom, Jakob Uszkoreit. “Scaling Autoregressive Video Models.” arXiv 1906.02634 (2019)]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441SJJGcaling ±utoregressive ;;8Video &&odels -- ´±¸330R 330RoIIFbot 00-ushing 107 [Dirk Weissenborn, Oscar Tackstrom, Jakob Uszkoreit. “Scaling Autoregressive Video Models.” arXiv 1906.02634 (2019)] 46IKUQO ;=SIKWLRWOPSRUIKO ;=XJLVKMIKOLQQ ;=PIKOO ;=SIKWLRWOPSRUIKO ;=XJLVKMIKOLQQ
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels 441SJJGcaling ±utoregressive ;;8Video &&odels -- $$inetiJJGcs 108 [Dirk Weissenborn, Oscar Tackstrom, Jakob Uszkoreit. “Scaling Autoregressive Video Models.” arXiv 1906.02634 (2019)] *RRNLQQ ±OOPW°WR°ULQRW JL\ OLNOOLRRRN² .XOO 3LQOWLKMV ±OOPW°WR°ULQRW JL\ OLNOOLRRRN²
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ''$atural ¸mage &&anipulation for ±utoregressive &&odels using ¹isher 441SJJGcores 109 [Wilson Yan, Jonatha Ho, Pieter Abbeel. “ “Natural Image Manipulation for Autoregressive Models using Fisher Scores.” arXiv 1912.05015 ■ &&ain JJGchallenge: ■ ³ow to get a latent representation from 00-ixel²''$''$? ■ <<9Why hard? 552The random input happens on a per-pixel sample IIFbasis ■ 00-roposed solution ■ 663Use ¹isher sJJGcore ''$ote: appliJJGcaIIFble to any likelihood model
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ''$atural ¸mage &&anipulation for ±utoregressive &&odels using ¹isher 441SJJGcores 110 [Wilson Yan, Jonatha Ho, Pieter Abbeel. ““Natural Image Manipulation for Autoregressive Models using Fisher Scores.” arXiv 1912.05015
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ''$atural ¸mage &&anipulation for ±utoregressive &&odels using ¹isher 441SJJGcores 111 [Wilson Yan, Jonatha Ho, Pieter Abbeel. ““Natural Image Manipulation for Autoregressive Models using Fisher Scores.” arXiv 1912.05015
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ´iIIFbliography JJGchar-rnn: http://karpathy.githuIIFb.io/2015/05/21/rnn-effeJJGctiveness/ &&±µ·: Germain, Mathieu, et al. "Made: Masked autoencoder for distribution estimation." .30TSTYZYKJWXWTSTHGFYZYONOUTUTSTHGFRQR (-UTUTSTLKKJWXWKJTSTIHKJ UTUTST 74HGFIHNMONOTSTKJ 63KJHGFWXWTSTONOTSTML . 2015. WaveNet: Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." HGFWXW?>ONO[\[\ VUVUWXWKJVUVUWXWONOTSTYZY HGFWXW?>ONO[\[\*$!& )° #$)) (2016). PixelCNN: Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." HGFWXW?>ONO[\[\ VUVUWXWKJVUVUWXWONOTSTYZY HGFWXW?>ONO[\[\*$!& !° &'%) (2016). Gated PixelCNN: Van den Oord, Aaron, et al. "Conditional image generation with pixelcnn decoders." &+JI[\[\HGFTSTIHKJXYX ONOTST 5KJZ[ZWXWHGFRQR .30TSTLKUTUWXWSRSHGFYZYONOUTUTST 7WXWUTUIHKJXYXXYXONOTSTML :9^_^_XYXYZYKJSRSXYX . 2016. PixelCNN++: Salimans, Tim, et al. "Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications." HGFWXW?>ONO[\[\ VUVUWXWKJVUVUWXWONOTSTYZY HGFWXW?>ONO[\[\*$!' !° %%!' (2017) Self-attention: Vaswani, Ashish, et al. "Attention is all you need." &+JI[\[\HGFTSTIHKJXYX ONOTST 5KJZ[ZWXWHGFRQR .30TSTLKUTUWXWSRSHGFYZYONOUTUTST 7WXWUTUIHKJXYXXYXONOTSTML :9^_^_XYXYZYKJSRSXYX . 2017. PixelSNAIL: Chen, Xi, et al. "Pixelsnail: An improved autoregressive generative model." HGFWXW?>ONO[\[\ VUVUWXWKJVUVUWXWONOTSTYZY HGFWXW?>ONO[\[\*$!'!"° )'&# (2017) Fast PixelCNN++: Ramachandran, Prajit, et al. "Fast generation for convolutional autoregressive models." HGFWXW?>ONO[\[\ VUVUWXWKJVUVUWXWONOTSTYZY HGFWXW?>ONO[\[\*$!' $° &  ! (2017). Multiscale PixelCNN: Reed, Scott, et al. "Parallel multiscale autoregressive density estimation." 7WXWUTUIHKJKJJIONOTSTMLXYX UTULK YZYNMKJ #$YZYNM .30TSTYZYKJWXWTSTHGFYZYONOUTUTSTHGFRQR (-UTUTSTLKKJWXWKJTSTIHKJ UTUTST 74HGFIHNMONOTSTKJ 63KJHGFWXWTSTONOTSTML±=<UTURQRZ[ZSRSKJ '  . JMLR. org, 2017. Grayscale PixelCNN: Kolesnikov, Alexander, and Christoph H. Lampert. "PixelCNN models with auxiliary variables for natural image modeling." 7WXWUTUIHKJKJJIONOTSTMLXYX UTULK YZYNMKJ #$YZYNM .30TSTYZYKJWXWTSTHGFYZYONOUTUTSTHGFRQR (-UTUTSTLKKJWXWKJTSTIHKJ UTUTST 74HGFIHNMONOTSTKJ 63KJHGFWXWTSTONOTSTML±=<UTURQRZ[ZSRSKJ '  . JMLR. org, 2017. Subscale Pixel Network: Menick, Jacob, and Nal Kalchbrenner. "Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling." HGFWXW?>ONO[\[\ VUVUWXWKJVUVUWXWONOTSTYZY HGFWXW?>ONO[\[\*$!(!"° !& ( (2018) Dirk Weissenborn, Oscar Tackstrom, Jakob Uszkoreit. “ Scaling Autoregressive Video Models.” arXiv 1906.02634 (2019) Sparse Attention: Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever. “Generating Long Sequences with Sparse Transformers.” arXiv 1904.10509 Wilson Yan, Jonathan Ho, Pieter Abbeel. “Natural Image Manipulation for Autoregressive Models using Fisher Scores.” arXiv 1912.05015 PixelCNN Super Resolution: Dahl, Ryan, Mohammad Norouzi, and Jonathon Shlens. "Pixel recursive super resolution." 7WXWUTUIHKJKJJIONOTSTMLXYX UTULK YZYNMKJ .30*/,*/,*/, .30TSTYZYKJWXWTSTHGFYZYONOUTUTSTHGFRQR (-UTUTSTLKKJWXWKJTSTIHKJ UTUTST (-UTUSRSVUVUZ[ZYZYKJWXW =<ONOXYXONOUTUTST . 2017. Grayscale PixelCNN: Kolesnikov, Alexander, and Christoph H. Lampert. "PixelCNN models with auxiliary variables for natural image modeling." 7WXWUTUIHKJKJJIONOTSTMLXYX UTULK YZYNMKJ #$YZYNM .30TSTYZYKJWXWTSTHGFYZYONOUTUTSTHGFRQR (-UTUTSTLKKJWXWKJTSTIHKJ UTUTST 74HGFIHNMONOTSTKJ 63KJHGFWXWTSTONOTSTML±=<UTURQRZ[ZSRSKJ '  . JMLR. org, 2017. 112
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%2 ±utoregressive &&odels ²olaIIFb 113
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help

Related Documents

FinStateMarkSequence-Summary.pdf
ilovepdf_merged-7.pdf
Counting-Examples.pdf
Practice lab 1.docx
Lecture 9 Semi-Supervised Learning and Unsupervised Distribution Alignment.pdf
InequalitiesSummary.pdf
Lecture11-Tree-Structured-Indexing.pdf
Lecture 7 Self-Supervised Learning.pdf
Lecture 3 Flow Models.pdf
Lecture 11 Learning from Text -- Alec Radford.pdf
soen331a1.pdf
soen331_a5.pdf
Recommended textbooks for you
Text book image
Operations Research : Applications and Algorithms
Computer Science
ISBN:9780534380588
Author:Wayne L. Winston
Publisher:Brooks Cole
Text book image
Management Of Information Security
Computer Science
ISBN:9781337405713
Author:WHITMAN, Michael.
Publisher:Cengage Learning,
Text book image
Principles of Information Systems (MindTap Course...
Computer Science
ISBN:9781305971776
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
Text book image
Information Technology Project Management
Computer Science
ISBN:9781337101356
Author:Kathy Schwalbe
Publisher:Cengage Learning
Text book image
Fundamentals of Information Systems
Computer Science
ISBN:9781305082168
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
Text book image
Principles of Information Systems (MindTap Course...
Computer Science
ISBN:9781285867168
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
SEE MORE TEXTBOOKS
Recommended textbooks for you
Text book image
Operations Research : Applications and Algorithms
Computer Science
ISBN:9780534380588
Author:Wayne L. Winston
Publisher:Brooks Cole
Text book image
Management Of Information Security
Computer Science
ISBN:9781337405713
Author:WHITMAN, Michael.
Publisher:Cengage Learning,
Text book image
Principles of Information Systems (MindTap Course...
Computer Science
ISBN:9781305971776
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
Text book image
Information Technology Project Management
Computer Science
ISBN:9781337101356
Author:Kathy Schwalbe
Publisher:Cengage Learning
Text book image
Fundamentals of Information Systems
Computer Science
ISBN:9781305082168
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
Text book image
Principles of Information Systems (MindTap Course...
Computer Science
ISBN:9781285867168
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
SEE MORE TEXTBOOKS

Browse Popular Homework Q&A

Q: The following concentration vs time data was obtained for the reaction: A→Products. Determine the…
Q: okay can you also do part b now.
Q: Explain why transcription and translation cannot occur simultaneously in eukaryotes but in…
Q: Why is it necessary to inoculate the KIA medium all the way down to the bottom of the tube during…
Q: Consider the following equilibrium process at 686. °C: CO₂(g) + H₂(g) CO(g) + H₂O(g) The equilibrium…
Q: Find Vectors and Perform Vector Operations in R³ Let a = (0, – 5, 5) and 6 = (4, 2, — 5). Find – 2ā…
Q: Consider the reaction 2 HBr(g) + Cl₂(g) → 2 HCI(g) + Br₂(g) Using the standard thermodynamic data in…
Q: The equilibrium H₂(g) + Br₂(g) 2 HBr (g) is 2.180 × 10° at 730.0 °c. Starting with 5.20 moles of HBr…
Q: Not sure on this.
Q: Consider the insoluble compound Iron(II) hydroxide, Fe(OH)2. The iron(II) ion also forms a complex…
Q: What is the major organic product obtained from the following reaction? HO O D) IV O A) I OB) II O…
Q: Given that 5 is a zero of the polynomial function f(x) find the remaining zeros.…
Q: Given the following data, RuO4 (aq) + e RuO4(s) + e RuO² (aq) RuO (aq) calculate AG for the…
Q: lisposal of the machine on January 1 in each separate situation. The machine needed extensive…
Q: What is the decimal value of this 32-bit two’s complement number? 1111 1111 1111 1111 1111 1111 1111…
Q: PART 1 Complete the given table for the function y = (2)x. x -2 -1 0 1 2 y
Q: A concave spherical mirror has a radius of curvature of magnitude 20.6 cm. (a) Find the location of…
Q: Prove each of the following statements using mathematical induction. (b) n Prove that for any…
Q: Find an equation of the plane. The plane through the points  (0, 9, 9), (9, 0, 9), and (9, 9, 0)
Q: American Revolution fundamentally change American society?
Q: During which phase of the growth curve are bacteria most susceptible to antibiotics? Why?
Q: www t = 0 R₁ !X R2 www Vc +ric - C
Q: e the data in the following table, which lists drive-thru order accuracy at popular fast food…
Q: Create the exponential function in the form f (x) = a(b)" that fits the two data points given: (0,…
Q: 3.- Prove that the bound 2/2ª in Lemma 4.1 is the best possible bound by showing that, if x = 2-d-1…