The Makefile.am here should be copied to your project's API documentation directory, and the variables edited appropriately. See doc/setting-up.txt for documentation on setting up a project to use gtk-doc. See the GTK+ and libglade sources for example Makefile.am option settings.