Earle, ACSaxe, AMRosman, Benjamin S2018-05-232018-05-232018-04Earle, A.C., Saxe, A.M. and Rosman, B.S. 2018. Hierarchical subtask discovery with non-negative matrix factorization. Sixth International Conference on Learning Representations (ICLR2018), 30 April 2018 - 3 May 2018, Vancouver Convention Center, Vancouver, Canadahttps://iclr.cc/Conferences/2018/Schedule?type=Posterhttps://openreview.net/forum?id=ry80wMW0Whttp://hdl.handle.net/10204/10228Paper presented at the Sixth International Conference on Learning Representations (ICLR2018), 30 April 2018 - 3 May 2018, Vancouver Convention Center, Vancouver, CanadaHierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains. However, learning an appropriate hierarchical decomposition of a domain into subtasks remains a substantial challenge. We present a novel algorithm for subtask discovery, based on the recently introduced multitask linearly-solvable Markov decision process (MLMDP) framework. The MLMDP can perform never-before-seen tasks by representing them as a linear combination of a previously learned basis set of tasks. In this setting, the subtask discovery problem can naturally be posed as finding an optimal low-rank approximation of the set of tasks the agent will face in a domain. We use non-negative matrix factorization to discover this minimal basis set of tasks, and show that the technique learns intuitive decompositions in a variety of domains. Our method has several qualitatively desirable features: it is not limited to learning subtasks with single goal states, instead learning distributed patterns of preferred states; it learns qualitatively different hierarchical decompositions in the same domain depending on the ensemble of tasks the agent will face; and it may be straightforwardly iterated to obtain deeper hierarchical decompositions.enReinforcement learningSubtask discoveryLMDPsConference PresentationEarle, A., Saxe, A., & Rosman, B. S. (2018). Hierarchical subtask discovery with non-negative matrix factorization. http://hdl.handle.net/10204/10228Earle, AC, AM Saxe, and Benjamin S Rosman. "Hierarchical subtask discovery with non-negative matrix factorization." (2018): http://hdl.handle.net/10204/10228Earle A, Saxe A, Rosman BS, Hierarchical subtask discovery with non-negative matrix factorization; 2018. http://hdl.handle.net/10204/10228 .TY - Conference Presentation AU - Earle, AC AU - Saxe, AM AU - Rosman, Benjamin S AB - Hierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains. However, learning an appropriate hierarchical decomposition of a domain into subtasks remains a substantial challenge. We present a novel algorithm for subtask discovery, based on the recently introduced multitask linearly-solvable Markov decision process (MLMDP) framework. The MLMDP can perform never-before-seen tasks by representing them as a linear combination of a previously learned basis set of tasks. In this setting, the subtask discovery problem can naturally be posed as finding an optimal low-rank approximation of the set of tasks the agent will face in a domain. We use non-negative matrix factorization to discover this minimal basis set of tasks, and show that the technique learns intuitive decompositions in a variety of domains. Our method has several qualitatively desirable features: it is not limited to learning subtasks with single goal states, instead learning distributed patterns of preferred states; it learns qualitatively different hierarchical decompositions in the same domain depending on the ensemble of tasks the agent will face; and it may be straightforwardly iterated to obtain deeper hierarchical decompositions. DA - 2018-04 DB - ResearchSpace DP - CSIR KW - Reinforcement learning KW - Subtask discovery KW - LMDPs LK - https://researchspace.csir.co.za PY - 2018 T1 - Hierarchical subtask discovery with non-negative matrix factorization TI - Hierarchical subtask discovery with non-negative matrix factorization UR - http://hdl.handle.net/10204/10228 ER -