A patch in #1817124: Disable showing "XML sitemap" fieldset on Taxonomy term edit page for users without 'administer xmlsitemap' permission revealed an issue with the xmlsitemap_link_bundle_access() function.
as mentioned in http://drupal.org/node/1817124#comment-6825434, xmlsitemap_link_bundle_access($bundle_info) ends up calling user_access('administer menus'). The issue is 'administer menus' is not a permission that exists, it should be 'administer menu' (without the s) instead. So the function is always returning false.