Lecture 4 Latent Variable Models -- VAE

pdf

School

University of Cincinnati, Main Campus *

*We aren’t endorsed by this school

Course

OPTIMIZATI

Subject

Electrical Engineering

Date

Oct 30, 2023

Type

pdf

Pages

90

Uploaded by BrigadierGorillaMaster2190

Report
±412294-158 ²LNNNLNNNijf 6346`a]jsuijfLNNNirvVVVWjsLNNNJLL !%"LNNNA>?AAir`a]VVVW`a]RTTT 00-ieter ±IIFbIIFbeel, ==:Xi (00-eter) ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan 663U² ´erkeley !%"LNNNJGHJJltuirLNNN 4 !%"A>?AAltLNNN`a]lt ;89;A>?AAirVVVWA>?AAIFGII^_LNNN °bc_JLLLNNN^_js -- ;89;A>?AAirVVVWA>?AAltVVVWbc_`a]A>?AA^_ ³ultbc_´`a]JGHJJbc_JLLLNNNir (;89;³´)
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%atent ;;8VariaIIFble &&odels ■ ±utoregressive models + ·lows ■ ±ll random variaIIFbles are oIIFbserved ■ !%%atent ;;8VariaIIFble &&odels (!%%;;8V&&s): ■ 441Some random variaIIFbles are hidden - we do not get to oIIFbserve 2
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder <<9Why !%%atent ;;8VariaIIFble &&odels? ■ 441Simpler, lower-dimensional representations of data often possiIIFble ■ !%%atent variaIIFble models hold the promise of automatiJJGcally identifying those hidden representations 3 Background & Wood bench in a park Obj± @ ²x³y´ & Corgi³ red µ white
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder <<9Why !%%atent ;;8VariaIIFble &&odels? ■ ±330R models are slow to sample IIFbeJJGcause all pixels (oIIFbservation dims) are assumed to IIFbe dependent on eaJJGch other ■ <<9We JJGcan make part of oIIFbservation spaJJGce independent EecGYtYecQG ec sebQ a<tQct v<rY<DaQs ■ !%%atent variaIIFble models E<c have faster sampling IIFby exploiting statistiJJGcal patterns 4
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%atent ;;8VariaIIFble &&odels ■ 441Sometimes, it’s possiIIFble to design a latent variaIIFble model with an understanding of the JJGcausal proJJGcess that generates data ■ ¸n general, we don’t know what are the latent variaIIFbles and how they interaJJGct with oIIFbservations ■ &&ost popular models make little assumption aIIFbout what are the latent variaIIFbles ■ ´est way to speJJGcify latent variaIIFbles is still an aJJGctive area of researJJGch 5
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¶xample latent variaIIFble model 6
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%atent ;;8VariaIIFble &&odel 7 ■ 441Sample ■ ¶valuate likelihood ■ 552Train ■ 330Representation
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 8 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& ■ ;;8Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 9 ■ &&otivation Ŷ 523irA>?AAVVVW`a]VVVW`a]RTTT !%"A>?AAltLNNN`a]lt ;89;A>?AAirVVVWA>?AAIFGII^_LNNN °bc_JLLLNNN^_js (VVVW`a]JGHJJ^_uJLLVVVW`a]RTTT ;89;³´ A>?AA`a]JLL µ<9:<<³´) Ŷ )&IFGII\]LNNNJGHJJltVVVWvLNNN ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& ■ ;;8Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 552Training !%%atent ;;8VariaIIFble &&odels 10 ■ ))&IIFbjeJJGctive: ■ 441SJJGcenario 1: z JJGcan only take on a small numIIFber of values → exaJJGct oIIFbjeJJGctive traJJGctaIIFble ■ 441SJJGcenario 2: z JJGcan take on an impraJJGctiJJGcal numIIFber of values to enumerate → approximate ³ow aIIFbout optimizing p Z (z) ? = “learning the prior” and sometimes done [more later] z x
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 11 ■ &&otivation Ŷ 523irA>?AAVVVW`a]VVVW`a]RTTT !%"A>?AAltLNNN`a]lt ;89;A>?AAirVVVWA>?AAIFGII^_LNNN °bc_JLLLNNN^_js (VVVW`a]JGHJJ^_uJLLVVVW`a]RTTT ;89;³´ A>?AA`a]JLL µ<9:<<³´) Ŷ )&IFGII\]LNNNJGHJJltVVVWvLNNN Ŷ °xact Ŷ -2.rior 162Sampling Ŷ ±mportance 162Sampling Ŷ Variational !"'ower ²ound (V!"'²) / °vidence !"'ower ²&+'und (°!"'²&+') ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& ■ ;;8Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¶xaJJGct !%%ikelihood ))&IIFbjeJJGctive ´xA>?AA_`ijf^_LNNN : mixture of 3 ¹aussians, with uniform prior over JJGcomponents 552Training oIIFbjeJJGctive: 12
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 2-µ &&ixture of ¹aussians 13
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-rior 441Sampling &&ain idea: if z JJGcan take on many values → sample z → run 441StoJJGchastiJJGc ¹radient µesJJGcent (441S¹µ) on the approximate oIIFbjeJJGctive 14
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-rior 441Sampling -- ¶xample + ²hallenge ²onsider data in ''$ JJGclusters: 441Sampling z uniformly results in only in only 1/''$ terms IIFbeing useful. 301LNNNJGHJJA>?AA^_^_: z might JJGcorrespond to many high level properties, e.g., 100 high level properties, proIIFbaIIFbly of JJGcorreJJGct z for given x: 0.5 100 µjsjsuLNNN: <<9When going to higher dimensional data, it IIFbeJJGcomes near impossiIIFble to IIFbe luJJGcky enough that a sampled z is a good matJJGch for a data point x (i) 15
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce 441Sampling -- &&otivation 0-.irbc_IFGII^_LNNN_` jsLNNNltltVVVW`a]RTTT: <<9Want to JJGcompute ´ut: (1) hard to sample from and/or (2) samples from a are not very informative ¶xample of (2): ''$ote: our !%%atent ;;8VariaIIFble &&odel oIIFbjeJJGctive is also example of (2) 16
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce 441Sampling -- ±lgorithm ¶bc_ir_`u^_A>?AAltVVVWbc_`a] : → ²an sample from q to JJGcompute expeJJGctation w.r.t. p 17
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce 441Sampling for !%%atent ;;8VariaIIFble &&odel 523irA>?AAVVVW`a]VVVW`a]RTTT )&IFGII\]LNNNJGHJJltVVVWvLNNN: ·bc_bc_JLL ijfirbc_ijfbc_jsA>?AA^_ JLLVVVWjsltirVVVWIFGIIultVVVWbc_`a] lhq(z)? <<9We want samples JJGcompatiIIFble with x (i) → ³ow aIIFbout µjsjsuLNNN: not JJGclear how to sample from this distriIIFbution... 18
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce 441Sampling 00-roposal µistriIIFbution ·LNNN`a]LNNNirA>?AA^_ 0-.irVVVW`a]JGHJJVVVWijf^_LNNN bc_QSSS ;89;A>?AAirVVVWA>?AAltVVVWbc_`a]A>?AA^_ ³ijfijfirbc_A>?AAJGHJJSUUU: <<9We JJGcan’t direJJGctly use p we want 441So, instead, we propose a parameterized distriIIFbution q we know we JJGcan work with easily (in this JJGcase, sample from easily), and try to find a parameter setting that makes it as good as possiIIFble. ¶.g. find as JJGclose as possiIIFble to 19
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce 441Sampling 00-roposal µistriIIFbution ;89;A>?AAirVVVWA>?AAltVVVWbc_`a]A>?AA^_ ³ijfijfirbc_A>?AAJGHJJSUUU ltbc_ ¶VVVW`a]JLLVVVW`a]RTTT lhq: <<9We JJGcan’t direJJGctly use p we want 441So, instead, we propose a parameterized distriIIFbution q we know we JJGcan work with easily (in this JJGcase, sample from easily), and try to find a parameter setting that makes it as good as possiIIFble. ¶.g. find as JJGclose as possiIIFble to 20
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce 441Sampling 00-roposal µistriIIFbution → optimize to find q ''$ote: all needed quantities in the oIIFbjeJJGctive readily JJGcomputaIIFble 21
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ±mortized ¸nferenJJGce ·LNNN`a]LNNNirA>?AA^_ µJLLLNNNA>?AA bc_QSSS ³_`bc_irltVVVWzA>?AAltVVVWbc_`a]: if same inferenJJGce proIIFblem needs to IIFbe solved many times, JJGcan we parameterize a neural network to solve it? )&uir JGHJJA>?AAjsLNNN: for all x (i) we want to solve: ³_`bc_irltVVVWzLNNNJLL QSSSbc_ir_`u^_A>?AAltVVVWbc_`a]: 523irA>?AAJLLLNNN-bc_QSSSQSSS: + : faster, regularization; - : not as preJJGcise 22
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ±mortized ¸nferenJJGce ³_`bc_irltVVVWzLNNNJLL QSSSbc_ir_`u^_A>?AAltVVVWbc_`a]: 23 z x ´.RTTT: ¶quivalently: with
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce <<9Weighted ±uto¶nJJGcoder (¸<<9W±¶) ))&IIFbjeJJGctive: ±nd: → maximize term1 - term2 24
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V¸ as ¸mportanJJGce 441Sampling 25 ■ <<9We draw multiple z samples from q(z|x) and name them z i ■ µefine w i and !%% k : [Burda et al¶³ ·¸±¹]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce <<9Weighted ±uto¶nJJGcoder (¸<<9W±¶) 26 [Burda et al¶³ ·¸±¹]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¸mportanJJGce <<9Weighted ±uto¶nJJGcoder (¸<<9W±¶) 27 [´urda et al., 2015]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 28 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling Ŷ Variational !"'ower ²ound (V!"'²) / °vidence !"'ower ²&+'und (°!"'²&+') ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& ■ ;;8Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V!%%´: µerivation 1 (°ensen) [live derivation] 29
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V!%%´: µerivation 2 ( $$!%%) 30 441Same as with °ensen’s, IIFbut now we know the gap = $$!%%
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8Variational !%%ower ´ound (;;8V!%%´) ■ <<9We now have an oIIFbjeJJGctive amenaIIFble to stoJJGchastiJJGc optimization ■ 552Turns out we JJGcan get more out of this exerJJGcise ■ note: the optimal q x (z) of ;;8V!%%´ is p(z|x), at whiJJGch point ;;8V!%%´ is tight (= log p(x)) 31
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V!%%´ &&aximization ■ ¹iven a data distriIIFbution x ~ p data , we JJGcan know train the generative model IIFby maximizing the ;;8V!%%´ under data distriIIFbution 32
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 33 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) Ŷ &+'ptimization Ŷ !"'ikelihood 051Ratio ³radients vs. 051Reparameterization 273Trick ³radients Ŷ &+'ptimizing the V!"'²/°!"'²&+' ■ ;;8Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient 34 [live derivation -- for general JJGcase; for ¹aussian of next slide]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient - 552Toy 00-roIIFblem 35 !%%earn to minimize the oIIFbjeJJGctive IIFbelow to reaJJGch the green point
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient - 552Toy 00-roIIFblem 36
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient - 552Toy 00-roIIFblem 37
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient - 552Toy 00-roIIFblem 38
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient - 552Toy 00-roIIFblem 39
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient - 552Toy 00-roIIFblem 40
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¹radient - 552Toy 00-roIIFblem 41
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative aka 330Reparameterization 552TriJJGck 42 [live derivation -- for general JJGcase; for ¹aussian of next slide]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative (00-µ) ■ 441StoJJGchastiJJGc gradient possiIIFble if z is JJGcontinuous now (more teJJGchniJJGcal JJGcondition?) ■ ²ommon JJGchoiJJGce: \eps ~ ''$ormal, f(\eps) = \mu + \sigma \eps ■ ±ny flow that you just learned! ■ ±lso known as reparameterization triJJGck ■ ²an work with only 1~2 samples 43
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative (00-µ) ■ 441Suppose we have some fixed noise sourJJGce \eps, and we JJGcan reparametrize z = f(\eps; \phi) [ \grad ¶_{z ~ q} [.. ] = ¶_{eps} [ \grad s(f(\eps; \phi)) ] ] 44
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 45 !%%earn to minimize the oIIFbjeJJGctive IIFbelow to reaJJGch the green point
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 46
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 47
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 48
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 49
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 50
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 51
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-athwise µerivative - 552Toy 00-roIIFblem 52
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V±¶ and !%%ikelihood 330Ratio ¹radient 53
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder !%%ikelihood 330Ratio ¶stimator ¸ssue : ³igh varianJJGce gradients, needs many samples of z to form a good estimate 54
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V±¶ and 00-athwise µerivative 55 [empty slide for live derivation what’s on next slide]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-µ applied to ;;8V¸ ■ ¸f we use JJGcontinuous z, then we JJGcan have direJJGctly maximize the latent variaIIFble model’s vlIIFb ■ ¸f p(z) = ''$ (z; 0, ¸), q(z|x) = ''$ (z; µ(x), diag( σ (x)^2)) ■ vlIIFb = ¶_{\eps ~ ''$} [ log p_\theta(x|z) + log p(z) - log q(\eps * σ _\phi(x)^2 + µ_\phi(x) ] ■ >>;You JJGcan now oIIFbtain stoJJGchastiJJGc gradient w.r.t. IIFboth \theta and \phi easily! ■ 552This is a ;;8Variational ±uto-¶nJJGcoder (;;8V±¶) 56
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V±¶ 57
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V±¶ 58
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V±¶ 59
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ²ompared to ±330R ■ <<9We now have a family of trainaIIFble latent variaIIFble models! ■ ´ut performanJJGce is laJJGcking 60
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder <<9Why is it JJGcalled an autoenJJGcoder? 61 ¸irLNNNA>?AA]^VVVW`a]RTTT JLLbc_w`a] ltSUUULNNN ´!%"¸)& / ;89;!%"¸ ■ <<9We have seen that a variational autoenJJGcoder is a latent variaIIFble model with ¹aussian prior p(z) and approximate posterior q(z|x). ■ <<9Why is it JJGcalled an “autoenJJGcoder”?
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 62 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& Ŷ Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 63 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& Ŷ Variations: Ŷ 162S&+'273T´: V/40Q-V´°, V/40Q-V´° 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ 64
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ -- ¶xperiments 65
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ -- ¶xperiments 66
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ 2.0 67
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ 2.0 -- ¶xperiments 68
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ 2.0 -- ¶xperiments ;;8V22/Q-;;8V±¶ 2.0 69 ´ig¹±''$ µeep
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ 2.0 -- ¶xperiments ;;8V22/Q-;;8V±¶ 2.0 70 ´ig¹±''$ µeep
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8V22/Q-;;8V±¶ 2.0 -- ¶xperiments ;;8V22/Q-;;8V±¶ 2.0 71 ´ig¹±''$ µeep
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 72 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& Ŷ Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 Ŷ ´051R + V´°: Variational !"'ossy ´uto°ncoder, -2.ixelV´° ■ µisentanglement: ´eta ;;8V±¶ ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder µeJJGcoder distriIIFbution ■ 441So far all models use simple distriIIFbution for p(x|z) ■ µue to laJJGck of expressivity itself, all entropy is pushed to z and z needs to JJGconvey a lot of information 73
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-owerful deJJGcoder ■ <<9What’s the maximum ;;8V!%%´? ■ <<9What if p(x|z) = p data (x)? ■ q(z|x) would IIFbe set to p(z) -> z has no information 74
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 00-owerful deJJGcoder ■ ³aving information in z inJJGcurs vlIIFb penalty of $$!%%(q ǁ p) whiJJGch is usually non-zero ■ “¸gnoring latent JJGcode” proIIFblems well doJJGcumented in literature ■ (·aIIFbius & van ±mersfoort,2014; ²hung et al., 2015; ´owman et al., 2015; 441SerIIFban et al., 2016; ·raJJGcJJGcaro et al., 2016; ==:Xu &441Sun, 2016) ■ &&any proposed solutions 75
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder <<9Weakening models ■ ±dding dropout in autoregressive JJGconditioning (´owman et al., 2015) ■ 00-ixel²''$''$ with limited reJJGceptive field (²hen et al., 2016) ■ ²onstant IIFbit rate µ !%% (q φ (z|x) ǁ p θ (z)) = JJGc (¹uu et al., 2017), (==:Xu & µurrett, 2018), (µavidson et al., 2018) ■ &&inimum IIFbit rate µ !%% (q φ (z|x) ǁ p θ (z)) ≥ δ (330Razavi et al., 2019) 76
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ²hanging training dynamiJJGcs ■ µ !%% (q φ (z|x) ǁ p θ (z)) warmup (´owman et al., 2015); (>>;Yang et al., 2017); ( $$im et al., 2018); (¹ulrajani et al., 2016) ■ “·ree-IIFbits” ( $$ingma et al., 2016); (²hen et al., 2016) ■ &&ore training updates to q(z|x) (³e et al., 2019) 77
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 78 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& Ŷ Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ Ŷ µisentanglement: ²eta V´° ■ 330Related ideas: ■ ;;8Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ´eta ;;8V±¶ 552The ´eta-;;8V±¶ oIIFbjeJJGctive is identiJJGcal to the ;;8V±¶ oIIFbjeJJGctive when IIFbeta = 1 79
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ´eta ;;8V±¶ -- ¶xperiments 80
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 81 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& ■ ;;8Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ Ŷ 051Related ideas: Ŷ Variational µequantization (flow++) ■ &&utual ¸nformation ¶stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder 330ReJJGcap: 663Uniform µequantization ■ 6346`a]VVVWQSSSbc_ir_` ²LNNNlhquA>?AA`a]ltVVVWzA>?AAltVVVWbc_`a] . ±dd noise to data. ■ ■ <<9We draw noise u uniformly from 82 [Theis³ Oord³ Bethge³ ·¸±º]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8Variational µequantization 83 ■ ;89;A>?AAirVVVWA>?AAltVVVWbc_`a]A>?AA^_ ²LNNNlhquA>?AA`a]ltVVVWzA>?AAltVVVWbc_`a] . ±dd a learnaIIFble noise q to data. [Ho et al¶³ ·¸±"]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ;;8Variational µequantization on ²¸·±330R 84 [Ho et al¶³ ·¸±"]
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ))&utline 85 ■ &&otivation ■ 552Training !%%atent ;;8VariaIIFble &&odels (inJJGcluding ;;8V±¶ and ¸<<9W±¶) ■ ))&IIFbjeJJGctive ■ ¶xaJJGct ■ 00-rior 441Sampling ■ ¸mportanJJGce 441Sampling ■ ;;8Variational !%%ower ´ound (;;8V!%%´) / ¶videnJJGce !%%ower ´))&und (¶!%%´))&) ■ ))&ptimization ■ !%%ikelihood 330Ratio ¹radients vs. 330Reparameterization 552TriJJGck ¹radients ■ ))&ptimizing the ;;8V!%%´/¶!%%´))& ■ ;;8Variations: ■ 441S))&552T±: ;;8V22/Q-;;8V±¶, ;;8V22/Q-;;8V±¶ 2.0 ■ ±330R + ;;8V±¶: ;;8Variational !%%ossy ±uto¶nJJGcoder, 00-ixel;;8V±¶ ■ µisentanglement: ´eta ;;8V±¶ Ŷ 051Related ideas: ■ ;;8Variational µequantization (flow++) Ŷ #($utual ±nformation °stimation
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder &&utual ¸nformation ■ &&utual information IIFbetween two random variaIIFbles ==:X, >>;Y: ¸(==:X; >>;Y) is defined as 86
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder &&utual ¸nformation ■ &&utual ¸nformation is a general way to measure dependenJJGcy IIFbetween two random variaIIFbles ■ 663Unlike the more JJGcommonly used JJGcovarianJJGce 87
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder &&utual ¸nformation ■ 663Useful in a lot of settings where one wants to maximize dependenJJGcy IIFbetween two variaIIFbles or estimate their dependenJJGcies: ■ ;;8Variational ¸nformation &&aximisation for ¸ntrinsiJJGcally &&otivated 330ReinforJJGcement !%%earning ■ ¸nfo¹±''$ ■ ²00-² ■ ... 88
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ¶stimating &&utual ¸nformation ■ <<9We JJGcan try to estimate the mutual information IIFbetween z and x in a latent variaIIFble model ■ ³as intraJJGctaIIFble posterior p(z|x) IIFbut we JJGcan estimate IIFby introduJJGcing a variational distriIIFbution q(z|x) 89
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help
663U² ´erkeley -- 441Spring 2020 -- µeep 663Unsupervised !%%earning -- 00-ieter ±IIFbIIFbeel, 00-eter ²hen, °onathan ³o, ±ravind 441Srinivas, ±lex !%%i, <<9Wilson >>;Yan -- !%%4 !%%atent ;;8VariaIIFble &&odels -- ;;8Variational ±uto¶nJJGcoder ´iIIFbliography ;;8V±¶: ±uto-¶nJJGcoding ;;8Variational ´ayes, µ. $$ingma and &&. <<9Welling, ¸²!%%330R 2014, https#»»arxiv¶org»pdf»±¼±·¶º±±½¶pdf ¸<<9W±¶: ¸mportanJJGce <<9Weighted ±utoenJJGcoders, >>;Y. ´urda, 330R. ¹rosse and 330R. 441Salakhutdinov, ¸²330R!%% 2015, https#»»arxiv¶org»pdf»±¹¸"¶¸¸¹±"¶pdf ;;8V22/Q-;;8V±¶: ''$eural µisJJGcrete 330Representation !%%earning, ±. van den ))&ord, ))&. ;;8Vinyals and $$. $$avukJJGcuoglu, ''$eur¸00-441S 2017, https#»»arxiv¶org»pdf»± ±±¶¸¸"¼ ¶pdf ;;8V22/Q-;;8V±¶ 2.0: ¹enerating µiverse ³igh-·idelity ¸mages with ;;8V22/Q-;;8V±¶-2, ±. 330Razavi, ±. van den ))&ord and ))&. ;;8Vinyals, ''$eur¸00-441S 2018, https#»»arxiv¶org»pdf»±"¸º¶¸¸½½º¶pdf ;;8V!%%±¶: ;;8Variational !%%ossy ±utoenJJGcoder, ==:X. ²hen et al, ¸²330R!%% 2017, https#»»arxiv¶org»pdf»±º±±¶¸· ¼±¶pdf 00-ixel;;8V±¶: ± !%%atent ;;8VariaIIFble &&odel for ''$atural ¸mages, ¸. ¹ulrajani et al, https#»»arxiv¶org»pdf»±º±±¶¸¹¸±¼¶pdf ¸±·-;;8V±¶: ¸mproving ;;8Variational ¸nferenJJGce with ¸nverse ±utoregressive ·low, µ. $$ingma et al, ''$eur¸00-441S 2016, https#»»arxiv¶org»pdf»±º¸º¶¸½"¼½¶pdf IIFbeta-;;8V±¶: !%%earning ´asiJJGc ;;8Visual ²onJJGcepts with a ²onstrained ;;8Variational ·ramework, ¸. ³iggins et al, ¸²!%%330R 2017,, https#»»openreview¶net»pdf?id&Sy·fzU"gl <<9Wake-441Sleep: 552The wake-sleep algorithm for unsupervised neural networks, ¹. ³inton et al, https#»»www¶cs¶toronto¶edu»~hinton»csc·¹¼¹»readings»ws¶pdf ;;8Variational µequantization (flow++) (·low++): ¸mproving ·low-´ased ¹enerative &&odels with ;;8Variational µequantization and ±rJJGchiteJJGcture µesign, °. ³o et al, ¸²&&!%% 2019, https#»»arxiv¶org»pdf»±"¸·¶¸¸· ¹¶pdf 90
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
  • Access to all documents
  • Unlimited textbook solutions
  • 24/7 expert homework help

Related Documents

IMG_8928.png
1028-FINAL-2023-SOL.pdf
Lecture 12 Representation Learning in Reinforcement Learning.pdf
1 Introduction .pdf
2 Introduction to MBSE.pdf
Hex to Bin to Dec Conversion.docx
L9 Project Testing.pdf
Sequencing_and_Phasing_PP_Notes.pdf
Netfun lab wk7.docx
Screenshot 2023-10-28 at 2.51.01 PM.png
note03.pdf
note12.pdf
Related Questions
Find the absolute and relative error of the following. a)The approximate value of e is used as 2.718. Calculate the absolute error and the relative error margin accordingly. b) P=3.1415927 and this value is given as an approximate result of p=3.1416. Calculate the absolute error and relative error margin accordingly.
I need answer for 5th question
can you solve this question handwritten solution step by step how did we obtain the v0 graph? my question is not a part of a graded assignment
hello there can u help for this question
A clamper circuit has 20 Vp-p. 100Hz square wave input voltage. The circuit consists of silicon diode IN4001 and 3V battery as shown in Figure 1 C. 0.1 µF D R Vi(t) 50 k2 Vo(t) 3 V Figure 1 a) Find the output voltage for all input voltages values. b) Sketch the output waveform, Vo(t).
SEE MORE QUESTIONS
Recommended textbooks for you
Text book image
Introductory Circuit Analysis (13th Edition)
Electrical Engineering
ISBN:9780133923605
Author:Robert L. Boylestad
Publisher:PEARSON
Text book image
Delmar's Standard Textbook Of Electricity
Electrical Engineering
ISBN:9781337900348
Author:Stephen L. Herman
Publisher:Cengage Learning
Text book image
Programmable Logic Controllers
Electrical Engineering
ISBN:9780073373843
Author:Frank D. Petruzella
Publisher:McGraw-Hill Education
Text book image
Fundamentals of Electric Circuits
Electrical Engineering
ISBN:9780078028229
Author:Charles K Alexander, Matthew Sadiku
Publisher:McGraw-Hill Education
Text book image
Electric Circuits. (11th Edition)
Electrical Engineering
ISBN:9780134746968
Author:James W. Nilsson, Susan Riedel
Publisher:PEARSON
Text book image
Engineering Electromagnetics
Electrical Engineering
ISBN:9780078028151
Author:Hayt, William H. (william Hart), Jr, BUCK, John A.
Publisher:Mcgraw-hill Education,
SEE MORE TEXTBOOKS
Related Questions
Recommended textbooks for you
Text book image
Introductory Circuit Analysis (13th Edition)
Electrical Engineering
ISBN:9780133923605
Author:Robert L. Boylestad
Publisher:PEARSON
Text book image
Delmar's Standard Textbook Of Electricity
Electrical Engineering
ISBN:9781337900348
Author:Stephen L. Herman
Publisher:Cengage Learning
Text book image
Programmable Logic Controllers
Electrical Engineering
ISBN:9780073373843
Author:Frank D. Petruzella
Publisher:McGraw-Hill Education
Text book image
Fundamentals of Electric Circuits
Electrical Engineering
ISBN:9780078028229
Author:Charles K Alexander, Matthew Sadiku
Publisher:McGraw-Hill Education
Text book image
Electric Circuits. (11th Edition)
Electrical Engineering
ISBN:9780134746968
Author:James W. Nilsson, Susan Riedel
Publisher:PEARSON
Text book image
Engineering Electromagnetics
Electrical Engineering
ISBN:9780078028151
Author:Hayt, William H. (william Hart), Jr, BUCK, John A.
Publisher:Mcgraw-hill Education,
SEE MORE TEXTBOOKS

Browse Popular Homework Q&A

Q: Given the reaction mechanism and the quantities listed below, find the limiting reagent and the…
Q: Do political science classes require the same amount of writing as history classes? The 59 randomly…
Q: What is the molality when 28.3 g of CO₂ (44.01 g/mol) is dissolved in 592 g of water? (Answer to 2…
Q: 84.0 g Ta to mol
Q: Download E2 and build a model to predict the S&P 500 value. The p value for the independent variable…
Q: Solubility of Ionic Compounds Classify each of the compounds as soluble or not soluble: iron(II)…
Q: A drug response curve describes the level of medication in the bloodstream after a drug is…
Q: Identify whether the variable is continuous or discrete: Occupation.
Q: What does this line of code mean in Matlab fprintf(‘%5s%20s%20s\n’)
Q: graph is decreasing/increasing
Q: The figure to the right shows the market demand for electricity and the average total cost and…
Q: In a certain city the temperature (in °F) t hours after 9 AM was modeled by the function T(t) = 44 +…
Q: Statistics Class Times A professor wishes to see if students show a time preference for statistics…
Q: Compute the total cost per year of the following pair of expenses. Then complete the sentence: On an…
Q: Molten carbonate fuel cells fueled with H₂ convert as much as 60% of the free energy released by the…
Q: Which of the following heterocycles is aromatic according to Huckel's rules? H-N: HIZ: II, III and…
Q: a. the numbers that Max could use to label each end of his number line to show the pairs of numbers…
Q: My calculated value of acceleration is 12.2 m/s^2 when I drop the object from a height of 3 m. And I…
Q: A chart used to accent the size of change is called:
Q: Use the References to access important values if needed for this question. When 33.22 is added to…
Q: Using the inner product (, ) : C( [0, 1] ) × C'( [0, 1] ) → R, (f, g) = √ 5 3 2 2 • ( ²2 ( - - ² ),…
Q: Problem 2: A cylinder of diameter D, length Lis immersed in a fluid that flows perpendicularly to…
Q: Calculate the Mean Absolute Deviation for the following data: Actual Forecast 94 90.3 [T 85 88.2 90…
Q: What is the mass in grams of each elemental sample? 0.589 molmol Ba to g
Q: Determine the time of concentration for use in the NRCS method for a drainage basin located in…