Documentation Generators Internals of Doxygen John Tully

Documentation Generators: Internals of Doxygen John Tully

Motivation: Why document code? n Required by company / administrators who examine code n Typically in software engineering, code isn’t consistently worked on n Co-workers (or “smart” users) look at source code for a better understanding of functionality n Code modified/built upon by others

Motivation: Why Automated? n Docs much more likely to be up to date - developers only need to update comments in code (editing word docs/latex files isn’t enjoyable) n Reuse of comments = ½ the work n Auto formatting n n More importantly, automatic cross-referencing if the automated system is sophisticated Huge advantage: what if recipients want different formats? n n n Management team (or dumb readers) End users Software testers or other software developers

Problem / Goals n Create a simple, easy-to-use, portable, highly configurable automatic documentation generator for a variety of output formats n Real motivation: Creator (Dimitri Van Heesch) used Qt, an application development framework (popular for creating GUIs) Wrote docs by hand to look exactly like Qt documentation; tried doc++ to do this automatically (not configurable enough) n source: sourceforge

Other Doc Generators n Several generators support more languages than Doxygen * n n n ROBODoc, Twin. Text, Natural Docs VB, Pascal, . NET, Perl, Java. Script, SQL Many (including Doxygen) allow addition of new languages n Discussion – ease use of these tools? n As far as output formats, Doxygen most versatile n Also seems to be best for diagram generation n n Dependency graphs, inheritance / collaboration diagrams Certain tools may do some things that Doxygen can’t (control flow, data flow); but they’re language-specific * Source for info on other generators: Wikipedia

Doxygen Information Flow external docs Doxywizard GUI Source Code Doxyfile (Config file) Doxytag Program HTML Pages Man Pages Tag file RTF Files MS-Word formatted Latex Files ps or pdf Doxygen Custom Stuff: - Headers - Footers - Images XML Source: Doxygen “Getting Started” Page dox xml parser (for custom output)

1. Documentation external docs Doxywizard GUI Source Code Doxyfile (Config file) Doxytag Program HTML Pages Man Pages Tag file RTF Files MS-Word formatted Latex Files ps or pdf Doxygen Custom Stuff: - Headers - Footers - Images XML dox xml parser (for custom output)

2. Configuration external docs Doxywizard GUI Source Code Doxyfile (Config file) Doxytag Program HTML Pages Man Pages Tag file RTF Files MS-Word formatted Latex Files ps or pdf Doxygen Custom Stuff: - Headers - Footers - Images XML dox xml parser (for custom output)

3. Parsing / Execution external docs Doxywizard GUI Source Code Doxyfile (Config file) Doxytag Program HTML Pages Man Pages Tag file RTF Files MS-Word formatted Latex Files ps or pdf Doxygen Custom Stuff: - Headers - Footers - Images XML dox xml parser (for custom output)

4. Output Options external docs Doxywizard GUI Source Code Doxyfile (Config file) Doxytag Program HTML Pages Man Pages Tag file RTF Files MS-Word formatted Latex Files ps or pdf Doxygen Custom Stuff: - Headers - Footers - Images XML dox xml parser (for custom output)

Doxygen Internals n Doxygen Information Flow: high-level view of steps taken by the user to document code n Doxygen Internals: overview of data flow as source files are parsed by Doxygen n Closer look at Parsing / Execution Phase mentioned previously

Doxygen Internals: Data Flow Config Parser C Preprocessor Language Parser Source: Doxygen“internals” page Documentation Parser Data Organizer Output Generators Source Parser n Config parser: processes configuration file n Written in flex n Simple (since config options are pretty simple – just 5 types) n C Preprocessor: fairly similar to standard preprocessor n Written in flex; uses yacc-based parser for expression evaluation n Language parser: convert input buffer into a tree of entries n Basically breaks up code into smaller modules that will be reorganized later

Doxygen Internals: Data Flow Config Parser C Preprocessor Language Parser Documentation Parser Data Organizer Output Generators Source Parser n Data organizer: Gather information from these entries n Build dictionaries of classes, files, variables, functions, groups, etc. n Determine relationships between these entities n Documentation parser: find comment blocks / tags in entities; feed results to output generators n Source parser (if source is documented): cross-referencing, syntax highlighting n Output generators: take data, which was found by language parser and organized/crossreferenced by data organizer, and generate output in specified format

Debugging Doxygen n For a thorough understanding of doxygen’s source code, understanding of flex important n n Lexical analyzer: generates scanners (programs that recognize lexical patterns in text) Executable produced from scanners, which execute C code when patterns are found n Flex has debugging option to output matched rules when they’re found – easy to follow the steps doxygen is taking n On internals page, a tool (script) is available which turns on debugging for any flex file

Limitations / Future Work n Limitations / “wish list” of doxygen discussed during demo n n More languages, more output formats, better template files Several improvements to internals also mentioned: n Language Parser: n n n One scanner / parser per language (currently one huge combined scanner) Modulate parsing (documentation blocks vs code) Parse preprocessor input for extended documentation (i. e. #defines)

Limitations / Future Work n Output generators – interesting future work Instead of using data structures (generated from the data organizer) to produce XML, use XML as an intermediate language n More info could be extracted by various output generators – with more understandable IL, easy to create better end tools n n Interactive source browsers n Configurable class diagram generators n Computing code metrics

Discussion