Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Example
Underlying implementation
The face class
( defclass face ( )
( ( name : i n i t a r g : name)
; ; P ro p e r t i e s :
( foreg r oun d : i n i t a r g : for egr o und )
( background : i n i t a r g : background )
( boldp : i n i t a r g : bold )
; ; et c .
( subfaces : i n i t a r g : subfaces ) ) )
6/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Example
A DSL for theme customization
How do we go from this. . .
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background bl ack
: face { o pt io n : fo reg ro und w hi te
: face { syn tax : bo ld t : fore gro und cyan }
: face { usage : f ore gro und ye ll ow }
}
. . . to that?
( setq default− th eme
( make−instance ’ fa ce : name ’ t o p l e vel
: background ’ b la ck
: subfa ces ( l i s t ( make− instance ’ f ace : name ’ o pt io n
: fo reg rou nd ’ white
: subfa ces ( l i s t ( make− instance ’ f ace : name ’ syn tax
: bold t
: fo reg rou nd ’ cyan )
( make−instance ’ fa ce : name ’ usage
: fo reg rou nd ’ ye ll ow ) ) ) ) ) )
7/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 1
Hook into the Lisp parser: reader macros
readtable: currently active syntax extensions table
macro character: special syntactic meaning
reader macro: implements macro character behavior
Let’s do it!
Make the {} characters active
Read a list of tokens until the closing brace
Push the symbol define-face on top of that list
Note: RTMP (Read-Time Meta-Programming)
9/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 1
Hook into the Lisp reader
This is how we go from this. . .
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background bl ack
: face { o pt io n : fo reg ro und w hi te
: face { syn tax : bo ld t : fore gro und cyan }
: face { usage : f ore gro und ye ll ow }
}
. . . to that:
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background bl ack
: face ( def ine − fac e o pt io n : fo reg ro und w hi te
: face ( def ine − fac e syn tax : bo ld t : fore gro und cyan )
: face ( def ine − fac e usage : f ore groun d ye ll ow ) )
10/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 2
Hook into the Lisp compiler: macros
Ordinary Lisp functions
Work on chunks of code (as data)
Transform expressions into new expressions
Control over evaluation
Let’s make define-face a macro!
Quoting its key arguments, except for the :face ones
Generating a call to make-face
Note: CTMP (Compile-Time Meta-Programming)
12/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 2
Hook into the Lisp compiler: macros
This is how we go from this. . .
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background bl ack
: face ( def ine − fac e o pt io n : fo reg ro und w hi te
: face ( def ine − fac e syn tax : bo ld t : fore gro und cyan )
: face ( def ine − fac e usage : f ore groun d ye ll ow ) )
. . . to that:
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background ’ b la ck
: face ( make−face ’ o pt io n : fo regro und ’ wh it e
: face ( make−face ’ syn tax : bo ld t : f ore gro und ’ cyan )
: face ( make−face ’ usage : f ore groun d ’ y el lo w ) )
13/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 3
A couple of wrappers
Lambda-list manipulation / 1st class functions
( defun make−face (name &r es t args &key &allow− other− ke ys )
( apply # ’ make−instance ’ f ace : name name ar gs ) )
( defun make−theme (& rest a rgs )
( apply # ’ make− face ’ t o p l ev e l args ) )
And while we’re at it. . .
( defmacro define− theme (& r est a rgs )
‘ ( def ine − fa ce t o p l e v el ,@args ) )
15/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 3
A couple of wrappers
This is how we go from this. . .
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background ’ b la ck
: face ( make−face ’ o pt io n : fo regro und ’ wh it e
: face ( make−face ’ syn tax : bo ld t : f ore gro und ’ cyan )
: face ( make−face ’ usage : f ore groun d ’ y el lo w ) )
. . . to that:
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background ’ b la ck
: face ( make−instance ’ face : name ’ o pt ion : fore gro und ’ wh it e
: face ( make−instance ’ face : name ’ sy ntax : bo ld t : f ore groun d ’ cyan )
: face ( make−instance ’ face : name ’ usage : for egr ou nd ’ y el lo w ) )
16/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 4
Hook into the object system: the MOP
The CLOS Meta-Object Protocol (MOP)
CLOS itself is object-oriented
The CLOS MOP: a de facto implementation standard
The CLOS components (classes, methods etc.) are
(meta-)objects of some (meta-)classes
Generic functions, methods
( defmethod func ( ( arg1 c la ss 1 ) arg2 . . . )
body )
Methods are outside the classes (ordinary function calls)
Multiple dispatch (multi-methods)
18/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 4
Hook into the object system: the MOP
Object instantiation (make-instance) is a protocol
Slot initialization (initialize-instance) is a
generic function
Let’s extend it!
Provide our own method for the face class
Collect all :face arguments
call the next (standard) method with a new :subfaces
initarg
19/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Step 4
Hook into the object system: the MOP
This is how we go from this. . .
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background ’ b la ck
: face ( make−instance ’ face : name ’ o pt ion : fore gro und ’ wh it e
: face ( make−instance ’ face : name ’ sy ntax : bo ld t : f ore groun d ’ cyan )
: face ( make−instance ’ face : name ’ usage : for egr ou nd ’ y el lo w ) )
. . . to that:
; ; ; d e f au l t . c th −−− Per sona l d e f au l t theme f o r Clon
: background ’ b la ck
: subfa ces ( l i s t ( make− instance ’ f ace : name ’ o pt io n : fo regro und ’ wh it e
: subfa ces ( l i s t ( make− instance ’ f ace
: name ’ syn tax : bo ld t : fo re gro und ’ cyan )
( make−instance ’ fa ce
: name ’ usage : fo reg round ’ ye ll ow ) ) ) )
20/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Wrap Up
Using the DSL externally
Mostly a matter of read, compile etc.
( defun read− user− theme ( )
( wit h− o pen − fi le ( stream ( merge− pathnames ".faces" ( user− homedir−pathname ) ) )
( read ( make− concatenated− stream ( make− string−input− stream "(define-theme " )
stream
( make− string− input− stream ")" ) ) ) ) )
( defmacro make− user−theme (& o p t i on a l compile )
( i f compile
‘ ( f un ca ll ( compile n i l ( lambda ( ) , ( read−user−theme ) ) ) )
( read−user−theme ) ) )
22/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Wrap Up
Using the DSL internally
Mostly a matter of. . . Just-Do-It
TM
( setq default− th eme
( define− theme
: background bl ack
: face { o pt io n : fo reg ro und w hi te
: face { syn tax : bo ld t : fore gro und cyan }
: face { usage : f ore gro und ye ll ow }
}
) )
23/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Conclusion
Impact of GPL on DSL design and implementation
Key GPL aspect: extensibility
Embedded homogeneous approach
A single language
DSL infrastructure smaller
DSL both internal and external
Common Lisp
Functional, Imperative, Object-Oriented
MOP
CTMP (macros)
RTMP (reader macros)
read, eval, compile
25/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Internal vs External DSLs
[Kamin, 1998, Czarnecki et al., 2004]
Sub-optimal syntax ok but. . .
Not ok:
[Fowler, 2010]: “external DSLs have their own custom
syntax and you write a full parser to process them”
[Kamin, 1998, Czarnecki et al., 2004]: “a prerequisite
for embedding is that the syntax for the new language
be a subset of the syntax for the host language”
BTW, same disagreement at the semantic level (MOP)
Poor error reporting
Research: [Tratt, 2008]
Lisp: ? (but Cf. condition system & restarts)
27/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
Controversial aspects of extensibility
Dynamic typing
pros: end-user friendly
cons: run-time type errors / checking
Research: [Taha and Sheard, 1997]
Hybrid languages (Cf. Racket)
Lazy Evaluation
pros: infinite data structures, new control primitives etc.
cons: pure functional languages only
Lisp: laziness through macros (not as straightforward),
but side-effects for free, and still functional.
28/37
Extensibility
Didier Verna
Introduction
Example
Step 1
Step 2
Step 3
Step 4
Wrap Up
Conclusion
Discussion
The root of (Lisp) extensibility
Reflection
Introspection
Intercession
Implementation
By API
Inherent: “homoiconicity” [McIlroy, 1960, Kay, 1969]
Further distinction [Maes, 1987, Smith, 1984]
Structural Reflection (program)
Behavioral Reflection (language)
29/37
Extensibility
Didier Verna
Bibliography I
Ahson, S. I. and Lamba, S. S. (1985).
The use of Forth language in process control.
Computer Languages, 10:179–187.
Czarnecki, K., O Donnell, J., Striegnitz, J., and Taha, W.
(2004).
DSL implementation in MetaOCaml, Template Haskell,
and C++.
In Lengauer, C., Batory, D., Consel, C., and Odersky,
M., editors, Domain-Specific Program Generation,
volume 3016 of Lecture Notes in Computer Science,
pages 51–72. Springer Berlin / Heidelberg.
30/37
Extensibility
Didier Verna
Bibliography II
Denert, E., Ernst, G., and Wetzel, H. (1975).
Graphex68 graphical language features in algol 68.
Computers & Graphics, 1(2-3):195–202.
van Deursen, A., Klint, P., and Visser, J. (2000).
Domain-specific languages: an annotated bibliography.
SIGPLAN Notices, 35:26–36.
Elliott, C. (1999).
An embedded modeling language approach to
interactive 3D and multimedia animation.
IEEE Transactions on Software Engineering,
25:291–308.
31/37
Extensibility
Didier Verna
Bibliography III
Fleutot, F. and Tratt, L. (2007).
Contrasting compile-time meta-programming in Metalua
and Converge.
In Workshop on Dynamic Languages and Applications.
Fowler, M. (2005).
Language workbenches: The killer-app for domain
specific languages?
Fowler, M. (2010).
Domain Specific Languages.
Addison Wesley.
32/37
Extensibility
Didier Verna
Bibliography IV
Hudak, P. (1998).
Modular domain specific languages and tools.
In Proceedings of the 5th International Conference on
Software Reuse, ICSR’98, pages 134–142,
Washington, DC, USA. IEEE Computer Society.
Kamin, S. N. (1998).
Research on domain-specific embedded languages and
program generators.
In Electronic Notes in Theoretical Computer Science,
volume 14. Elsevier.
Kay, A. C. (1969).
The Reactive Engine.
PhD thesis, University of Hamburg.
33/37
Extensibility
Didier Verna
Bibliography V
Maes, P. (1987).
Concepts and experiments in computational reflection.
In OOPSLA. ACM.
McIlroy, M. D. (1960).
Macro instruction extensions of compiler languages.
Commun. ACM, 3:214–220.
McNamara, B. and Smaragdakis, Y. (2000).
Functional programming in C++.
SIGPLAN Not., 35:118–129.
Pagan, F. (1979).
ALGOL 68 as a metalanguage for denotational
semantics.
The Computer Journal, 22(1):63–66.
34/37
Extensibility
Didier Verna
Bibliography VI
Prud’homme, C. (2006).
A domain specific embedded language in C++ for
automatic differentiation, projection, integration and
variational formulations.
Journal of Scientific Programming, 14:81–110.
Rompf, T., Sujeeth, A. K., Lee, H., Brown, K. J., Chafi,
H., Odersky, M., and Olukotun, K. (2011).
Building-blocks for performance oriented DSLs.
In DSL’11: IFIP Working Conference on
Domain-Specific Languages.
Skalski, K., Moskal, M., and Olszta, P. (2004).
Meta-programming in Nemerle.
Technical report.
35/37
Extensibility
Didier Verna
Bibliography VII
Smith, B. C. (1984).
Reflection and semantics in Lisp.
In Symposium on Principles of Programming
Languages, pages 23–35. ACM.
Taha, W. and Sheard, T. (1997).
Multi-stage programming with explicit annotations.
In Proceedings of the 1997 ACM SIGPLAN symposium
on Partial evaluation and semantics-based program
manipulation, PEPM ’97, pages 203–217, New York,
NY, USA. ACM.
36/37
Extensibility
Didier Verna
Bibliography VIII
Tratt, L. (2008).
Domain specific language implementation via
compile-time meta-programming.
ACM Transactions on Programming Languages and
Systems, 30:31:1–31:40.
Tratt, L. (2005).
Compile-time meta-programming in a dynamically typed
OO language.
Vasudevan, N. and Tratt, L. (2011).
Comparative study of DSL tools.
Electronic Notes in Theoretical Computer Science,
264:103–121.
37/37
