{"id":14233,"date":"2024-03-17T23:06:21","date_gmt":"2024-03-17T23:06:21","guid":{"rendered":"\/?p=14233"},"modified":"2024-03-17T23:12:22","modified_gmt":"2024-03-17T23:12:22","slug":"mixture-of-experts","status":"publish","type":"post","link":"\/?p=14233","title":{"rendered":"Mixture of Experts"},"content":{"rendered":"

Omar Sanseviero @osanseviero Grok weights are out. Download them quickly<\/em> at https:\/\/huggingface.co\/xai-org\/grok-1<\/p>\n

huggingface-cli download xai-org\/grok-1 –repo-type model –include ckpt\/tensor* –local-dir checkpoints\/ckpt-0 –local-dir-use-symlinks False<\/p>\n

Learn about mixture of experts at https:\/\/hf.co\/blog\/moe
\nReplying to @osanseviero<\/p>\n

\n
\n
It seems there is a conflict between saying “Grok-1 <\/span>open<\/strong>-weights model” and “Due to the large size of the model (314B parameters), a multi-GPU machine is required” on the same page. Your people need to learn that “open data” means it has to be accessible to all. A few weights at a time, statistical summaries, examples, open description, links and background, some effort at an open community. “Here, we dumped this on the web. See, we shared!!” “You have no way to verify it, because you do not have the big computers that we have!!”<\/span><\/div>\n<\/div>\n<\/div>\n
\n
\n
\u00a0<\/span><\/div>\n<\/div>\n<\/div>\n
\n
\n
\n

Your “Mixture of Experts Explained” is very revealing. Yes, I looked at your “Open Source MoEs” project links.<\/span><\/p>\n

Are those few groups with big computers supposed to be the “Mixture of Experts”? It is not that hard if you actually share and use global open tokens.\u00a0 When I see the flopping door on the suborbital test, I know why you do things the way you do. Tell him to stop saying “open”.<\/p>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\u00a0<\/span><\/div>\n<\/div>\n<\/div>\n
\n
\n
Do you know that “Moe” in Japanese means “feelings of strong affection toward characters in anime, manga, video games, and other media – directed at the otaku markets” So I recommend you always write out “Mixture of Experts approach”<\/span><\/div>\n<\/div>\n<\/div>\n

Richard Collins, The Internet Foundation<\/p>\n","protected":false},"excerpt":{"rendered":"

Omar Sanseviero @osanseviero Grok weights are out. Download them quickly at https:\/\/huggingface.co\/xai-org\/grok-1 huggingface-cli download xai-org\/grok-1 –repo-type model –include ckpt\/tensor* –local-dir checkpoints\/ckpt-0 –local-dir-use-symlinks False Learn about mixture of experts at https:\/\/hf.co\/blog\/moe Replying to @osanseviero It seems there is a conflict between saying “Grok-1 open-weights model” and “Due to the large size of the model (314B parameters),
Read More »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[43],"tags":[],"class_list":["post-14233","post","type-post","status-publish","format-standard","hentry","category-assistive-technologies"],"_links":{"self":[{"href":"\/index.php?rest_route=\/wp\/v2\/posts\/14233","targetHints":{"allow":["GET"]}}],"collection":[{"href":"\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=14233"}],"version-history":[{"count":5,"href":"\/index.php?rest_route=\/wp\/v2\/posts\/14233\/revisions"}],"predecessor-version":[{"id":14238,"href":"\/index.php?rest_route=\/wp\/v2\/posts\/14233\/revisions\/14238"}],"wp:attachment":[{"href":"\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=14233"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=14233"},{"taxonomy":"post_tag","embeddable":true,"href":"\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=14233"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}