Introduction
Overview
Demo
Challenges Conclusion
Introduction
I
Common Lisp: Social / Community Aspects
I
The most expressive and extensible language (homoiconicity)
I
Drawbacks: technical social challenges
I
Individualism
I
(Too) Many dierent solutions for every problem
I
Quality dicult to assert
I
Many of them ad-hoc or 80% nished
I
Lack of documentation
I
Consolidation Eorts
I
Websites, Resources (guides, tutorials, wikis etc.)
I
ASDF, Quicklisp
I
Introducing Quickref
I
Global automatic documentation project for Quicklisp libraries
I
<don> Reference manuals 6= user manuals </don>
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 2/19
Introduction
Overview
Demo
Challenges Conclusion
Documentation Extraction
I
Distribution (README les etc.)
I
ASDF metadata (author, description, repository, etc.)
I
Language-level documentation (docstrings)
I
The rest (software components)
I
Code Walking (lightweight but very dicult)
I
Introspection (easier but requires loading)
system components, packages, constants, variables, macros,
functions, methods, structures, classes, types, etc.
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 6/19
Introduction
Overview
Demo
Challenges Conclusion
Documentation Extraction
I
Distribution (README les etc.)
I
ASDF metadata (author, description, repository, etc.)
I
Language-level documentation (docstrings)
I
The rest (software components)
I
Code Walking (lightweight but very dicult)
I
Introspection (easier but requires loading)
system components, packages, constants, variables, macros,
functions, methods, structures, classes, types, etc.
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 6/19
ASDF metadata
(asdf:defsystem :net.didierverna.declt
:long-name "Documentation Extractor from Common Lisp to Texinfo"
:description "A reference manual generator for Common Lisp libraries"
:author "Didier Verna"
:mailto "didier@didierverna.net"
:homepage "http://www.lrde.epita.fr/~didier/software/lisp/"
:source-control "https://github.com/didierverna/declt"
:license "BSD"
...)
Introduction
Overview
Demo
Challenges Conclusion
Documentation Extraction
I
Distribution (README les etc.)
I
ASDF metadata (author, description, repository, etc.)
I
Language-level documentation (docstrings)
I
The rest (software components)
I
Code Walking (lightweight but very dicult)
I
Introspection (easier but requires loading)
system components, packages, constants, variables, macros,
functions, methods, structures, classes, types, etc.
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 6/19
Documentation strings
(defmacro @defconstant (name &body body)
"Execute BODY within a @defvr {Constant} NAME environment.
NAME is escaped for Texinfo prior to rendering.
BODY should render on *standard-output*."
`(@defvr "Constant" ,name ,@body))
Introduction
Overview
Demo
Challenges Conclusion
Toolchain
Quicklisp
foo/
Declt
foo.texi
Makeinfo
foo.html
I
Why Texinfo?
I
Well suited to technical documentation (reference manual)
I
Easily converted (PDF, HTML, Info, etc.)
I
Built-in index / anchoring / cross-reference facility
I
Declt: Introspection
I
Compilation / loading (of dependencies) may be required
I
Avoid loading 2000 libraries in the same Lisp image!
I
Run Declt in external processes
I
Makeinfo: Perl/C script
I
Ditto
I
Quickref: Additional glue + loop over all Quicklisp libraries
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 7/19
Introduction
Overview
Demo
Challenges Conclusion
Performance
Library Pool
Declt
Declt
Declt
Texinfo Files Makeinfo
Makeinfo
Makeinfo
HTML Files
I
Sequential Processing
I
Absolute worst-case sequential scenario: 7h
I
Typical scenario: 2h
I
Parallel Processing + scheduling algorithm
I
4x performance improvement
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 8/19
Introduction
Overview
Demo
Challenges Conclusion
A Stress Test for Texinfo
I
Scalability
I
2000 or so libraries
I
Dependency management
I
Foreign dependencies
I
Library / documentation size
I
Texinfo Figures
I
File sizes: 7Ko – 15Mo (x2 HTML)
I
Lines of code: 364 – 285,020
I
Index entries: 14 – 44,500
I
Processing time: 0.3s – 1, 38s
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 12/19
Introduction
Overview
Demo
Challenges Conclusion
Metadata Format Underspecication
:author "Didier Verna"
:author "Didier Verna <didier@lrde.epita.fr>"
:author "Didier Verna didier@lrde.epita.fr"
:author "didier@lrde.epita.fr"
:author "<didier@lrde.epita.fr>"
:author "Didier Verna and Antoine Martin"
:author "Didier Verna, Antoine Martin"
:author "Didier Verna Antoine Martin"
:author "D. Verna Antoine E Martin"
:author "D. Verna Antoine \"Joe Cool\" Martin"
:author ("Didier Verna" "Antoine Martin")
I
Social incentive: people don’t like their work to look bad on my
public website…
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 13/19
Introduction
Overview
Demo
Challenges Conclusion
Metadata Format Underspecication
:author"
Original Authors:
Salvi Péter,
Naganuma Shigeta,
Tada Masashi,
Abe Yusuke,
Jianshi Huang,
Fujii Ryo,
Abe Seika,
Kuroda Hisao
Author Post MSI CLML Contribution:
Mike Maul <maul.mike@gmail.com>"
I
Social incentive: people don’t like their work to look bad on my
public website…
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 13/19
Introduction
Overview
Demo
Challenges Conclusion
Metadata Format Underspecication
:author "(let ((n \"Christoph-Simon Senjak\")) ~
(format nil \"~A <~C~C~C~C~A>\" ~
n (elt n 0) (elt n 10) (elt n 16) ~
#\\@ \"uxul.de\"))"
I
Social incentive: people don’t like their work to look bad on my
public website…
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 13/19
Introduction
Overview
Demo
Challenges Conclusion
Denitions Grouping
Example 1: accessors
context-hyperlinksp CONTEXT [Function]
(setf context-hyperlinksp) BOOL CONTEXT [Function]
Access CONTEXT ’s hyperlinksp ag.
Package [net.didierverna.declt], page 29,
Source [doc.lisp], page 24, (le)
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 14/19
Introduction
Overview
Demo
Challenges Conclusion
Denitions Grouping
Example 2: generic functions
document ITEM CONTEXT [Generic Function]
Render ITEM’s documentation in CONTEXT.
Package [net.didierverna.declt], page 29,
Source [doc.lisp], page 24, (le)
Methods
document (SYSTEM system) CONTEXT
Render SYSTEM’s documentation in CONTEXT.
Source [asdf.lisp], page 26, (le)
document (MODULE module) CONTEXT
Render MODULE’s documentation in CONTEXT.
Source [asdf.lisp], page 26, (le)
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 14/19
Introduction
Overview
Demo
Challenges Conclusion
Denitions Grouping
I
Only use the low level interface: @deffn, @defvr, etc.
I
Environment nesting → unwanted indentation
I
Heterogeneous @def... / @def...x prohibited
I
Heterogeneous categories authorized
@deffn {Function} ...
@deffnx {Compiler Macro} ...
I
Remaining Limitations
I
Only 9 xed canonical categories
I
Some distinctions arguable (e.g. typed vs. untyped)
I
Heterogeneous mixing still prohibited
@deffn {Function} foo ...
@defvrx {Symbol Macro} foo ...
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 14/19
Introduction
Overview
Demo
Challenges Conclusion
Pretty Printing
I
Names can be anything → escaping vs. “revealing”
I
|my stuff| vs. my␣stu
I
(setf foo) vs. (setf foo)
I
|argument(s)| vs. argument(s)
I
Symbol qualication: my.long.package.name:symbol
I
In general: avoid
I
Sometimes leads to ambiguous output (e.g. method specializers)
I
Docstrings: what to do? Verbatim, simple heuristic(s), markup etc.
I
References: @ref{anchor, , label} gives varying output
HTML label
PDF [label], page 12, → trailing comma (or not)
Info *note label: anchor.
Emacs See label. → Casing seems to vary
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 15/19
Introduction
Overview
Demo
Challenges Conclusion
Anchoring
I
Anchor names limitations (dots, commas, colons, parens)
I
Use <dot> etc. (ugly; use UTF8 characters instead?)
I
Anchor text less constrained, not well documented
I
Avoid nodes as much as possible…
I
Problems above
I
Uniqueness of names
I
No control over the display
I
…in particular, nodes associated with Lisp symbols
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 16/19
Introduction
Overview
Demo
Challenges Conclusion
Conclusion
I
A successful project
I
Almost 2000 libraries nicely documented
I
Less than 2% still cause problems
I
The community is grateful
I
A successful stress test for Texinfo
I
Reliable and scalable
I
Sometimes gets in the way, but still a good choice
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 18/19
Introduction
Overview
Demo
Challenges Conclusion
Perspectives
I
Casing
I
More / dierent indexes
I
More links / cross-references (external notably)
I
More / improved pretty-printing
I
Provide PDF & Info on the website as well
I
Emacs / Slime integration
I
More index (web) pages
I
…
Quickref: a Stress Test for Texinfo / TUG 2019 – Didier Verna 19/19
