Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
Don Knuth @ TUG 2010
Why did you design T
E
X as a macro-expansion based system?
1 Wanted something simple to use for my secretary
2 Computational resources at the time were limited
1 Is T
E
X simple to use, really?
2 Computational resources are not limited anymore
4/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
A better T
E
X?
What would that be?
T
E
X’s strength is in the quality of its typesetting, not in its
programmatic interface.
Keep the typesetting functionality but provide. . .
A more modern and consistent API
Real programming capabilities
Still simple to use (at least for simple things)
Extensibility / customizability
Backward Compatibility
5/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
Alternatives
eval4tex, perlT
E
X, QAT
E
X, PyT
E
X, python, sT
E
Xme, LuaT
E
X, iT
E
X. . .
Wrap T
E
X in a programming language
Wrap a programming language in T
E
X
Writing macros in another language
Getting rid of macros
Synchronous dual-process (std redirection / file I/O)
Multi-pass
What about a fully integrated approach?
I know, NTS is dead. . .
6/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
Why Common Lisp?
A language that doesn’t get in the way
Old language (= obsolete, = mature and modern)
ANSI standard (1994) ⇒ stable
Industrial-scale general purpose language
Multi-paradigm
Highly optimizable
Pletora of libraries
Scripting / extension language
Highly dynamic
Highly reflexive
Easy to learn (no syntax)
9/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
Built-in paradigms
Free of charge
key/value interface: functions lambda-lists
Packages: ASDF systems
Namespaces: Common Lisp packages
Interactive behavior: conditions and restarts
Dumping: Lisp images (idea: user-level dumping)
Performance:
Interpretation / Compilation / JIT-Compilation
Static typing
And again, dumping
. . .
10/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
Objectives
Remember them?
✗ A more modern and consistent API
✔ Real programming capabilities
✔ Still simple to use (at least for simple things)
✔ Extensibility / customizability
✗ Backward Compatibility
13/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
A more modern and consistent API
Programmatic T
E
X primitives
Parameters ⇒ Lisp variables
badness
Quantities ⇒ Lisp objects
(setf baselineskip #g(b :plus x :minus y))
Commands ⇒ Lisp functions
(input file)
(hbox material)
(hbox material :to dim)
(hbox material :spread dim)
(hbox-to dim material)
(hbox-spread dim material)
The typesetting subset of T
E
X
No \def, \relax and friends
14/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
Backward Compatibility
With good’old T
E
X
Implement traditional T
E
X on top of procedural T
E
X
(part of) T
E
X’s digestive engine
Provide a way to plug Lisp code in T
E
X files
T
E
X macros written in Lisp or direct Lisp code
15/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
What does it take to embed Lisp in T
E
X?
Provided T
E
X is written in Lisp, that is :-)
( in−p ackag e : com. d v l s o f t . t i c l )
( in −r ea dt abl e : com. d v l s o f t . t i c l )
; ; ; ; U t i l it i e s
( defun 2+ ( number )
"Like 1+ but add 2 instead."
(+ nu mber 2 ))
( defun 3+ ( number )
"Like 1+ but add 3 instead."
(+ nu mber 3 ))
; ; ; ; S t r i n g u t il i t i es
( defun hex− char− p ( char )
"Return T if CHAR is in 0123456789abcdef."
(member c har ’ (# \0 #\ 1 #\ 2 #\ 3 #\ 4 #\ 5 #\ 6 #\ 7 # \8 # \9
#\ a #\ b #\ c # \d # \ e #\ f ) ) )
( defun e ol− in dex ( st ri n g &o pt io na l ( s ta r t 0) )
"Return the next end-of-line index in STRING from START."
( po si t io n # \ Newl ine st ri n g : s t ar t st a rt ) )
( defun tr ai l− i n de x ( st r i ng & op ti on al ( s ta r t 0 )
( end ( or ( eo l−i nd ex s tr i n g st a rt )
( len gt h s t r in g ) )) )
"Return the next trail index in STRING between START and END."
( loop : wi th tr ai l −i n d e x : = n i l
: f or in dex : fro m (1− end) : down to s ta r t
: i f ( cha r= ( a re f st ri n g i nde x ) # \S pace )
: do ( s etq tr a i l− i n d ex in dex )
: el se
: re t ur n tr ai l− i nd e x
: f i n a ll y ( re tu r n t r a il − i n de x ) ) )
; ; ; ; TeX pro c es s in g
( de f st r uc t te x− s tr in g
st r i ng ; the s t r i n g bei ng pro ce sse d
le ng th ; t h e l e n gt h of STRING
( in dex 0) ; th e i n de x o f the ne xt ch a ra c t er to pro c es s i n STRING
t r a i l− i n d ex ; th e i nd e x o f t h e spa ce t r a i l o n th e cu r re n t l i n e in STRING
eo l−i nde x ; th e i n de x o f the ne xt new l in e c h a r ac t e r i n STRING
eo lp ; w he th er we have re ac hed the e nd o f t h e c u r r en t l i n e
( s ta te : N) ) ; t he cu r r e n t st a t e o f TeX
( defun te x− s tr in g
( st r in g &au x ( le ng th (l e ng th st ri n g ))
( eol −in dex ( eol −in dex str i n g ))
( t ra il −i nd e x ( t ra i l − in d e x s t r in g 0 ( or e ol− in dex l e ng th ) ) ) )
"Return a new processing context for STRING."
( mak e−te x−s tr ing : st r i ng st r in g
: le ng th l en g th
: t ra il −i n d ex t ra il −i n d e x
: eo l−i nd ex eo l−i nde x ))
( defun g et −t ex− l ine ( te x− s tr in g )
"Initiate the next line of TEX-STRING."
; ; #### PORTME : u s in g wi th − s l ot s on a s t r u c t .
( wi th −s lo ts ( st r i ng len gt h i nde x t ra il −i n d ex eo l−i nde x e ol p ) t ex −s tr in g
( unle ss ( = in dex len gt h )
( se t f
eo l−i nde x ( eo l−i nde x st r in g ind ex )
t r a i l− i n d ex ( t ra il −i n d ex s tr i ng in dex ( or e ol− in dex l en g th ) ) ) )
( se t f e ol p ni l ) ) )
( defun s ki p−t ex −li ne ( te x− s tr in g )
"Discard the rest of the current line in TEX-STRING."
( wi th −s lo ts (l en g th i nd ex e ol− i nd ex eo lp ) te x− st ri ng
( i f eo l−i nde x
( se t f i nd ex (1+ e ol− in dex )
eo lp t )
( se t f i nd ex len gt h )) ) )
( def va r endlinechar #\ Re tur n
"\\endlinechar.
The
character to insert at end of lines." )
# i ( get −te x−c ha r 1)
( defun g et −te x−c ha r ( t ex −s tr in g &o pt io na l lo oku p )
"Get the next TeX character from TEX-STRING.
Return either the next character on the current line, :EOL if the whole
current line has been read, or nil if there is nothing left to read.
If LOOKUP, don’t actually eat up the character in question."
; ; #### PORTME : u s in g wi th − s l ot s on a s t r u c t .
( wi th −s lo ts ( st r i ng len gt h i nde x t ra il −i n d ex eo l−i nde x e ol p ) t ex −s tr in g
( unle ss ( = in dex len gt h )
; ; #### NOTE: ev en when we j u s t LOOKUP th e n e xt ch a ra c t er wi t h o u t
; ; a c t u a l l y ea t in g i t , i t doe sn ’ t hu r t to sk i p t h e tr ai li n g sp ace s f o r
; ; r e a l .
(when ( eql ind ex tr ai l− i n de x )
( se t f i nd ex eol −in dex ) )
(when in de x
( cond ( eo lp
: eo l )
( ( eql i nd ex eol −in dex )
( pro g1 endlinechar
( unle ss l ook up
( in c f i nd ex )
( se t f e ol p t ) ) ) )
( (= ( ca tc od e ( ar ef st r in g ind ex ) ) + su pe r sc ri p t +)
( l e t ( ( char ( are f st ri n g i nd ex ) )
( l i mi t ( or eo l−i nde x len gt h )) )
( cond ( (and ( >= (− l i m it i nd ex ) 4 )
( char = char ( ar e f s t r in g ( 1+ i nd ex ) ) )
( hex−char −p ( ar e f s t r in g ( 2+ i nd ex ) ) )
( hex−char −p ( ar e f s t r in g ( 3+ i nd ex ) ) ) )
( pro g1 (c ode− cha r ( rea d−f rom −st rin g
( for mat n i l "#x~C~C"
( ar e f s t ri n g ( 2+ in de x ) )
( ar e f s t ri n g ( 3+ in de x ) ) ) ) )
( unle ss l ook up
( in c f i nd ex 4 )) ) )
( (and ( >= (− l i m it i nd ex ) 3 )
( char = char ( ar e f s t r in g ( 1+ i nd ex ) ) )
( < ( char −code ( ar e f s t ri n g ( 2+ i nd ex ) ) ) 12 8) )
( l e t ( ( char −co de
( char− cod e ( are f st ri n g ( 2+ in de x ) ) ) ) )
( pro g1 (c ode− cha r (+ cha r−c ode
( i f (< c har −co de 64 )
64
−6 4) ))
( unle ss l ook up
( in c f i nd ex 3 )) ) ) )
( t
( pro g1 ( ar e f s t ri n g i nd ex )
( unle ss l ook up
( in c f i nd ex ) ) ) ) ) ))
( (= ( cat cod e ( ar e f s t ri n g i nd ex ) ) + su bs cr ip t +)
( l e t ( ( char ( are f st ri n g i nd ex ) )
( l i mi t ( or eo l−i nde x len gt h )) )
( cond ( (and ( >= (− l i m it i nd ex ) 2 )
( char = char ( ar e f s t r in g ( 1+ i nd ex ) ) ) )
( mu lti pl e− v al ue −b ind ( ob je ct n ew−i ndex )
( re ad −fr om− st rin g st r i ng t n il
: s t a r t (2 + in de x )
: pr es erv e− wh ite sp ace t )
( l e t ( ( ne w− st rin g ( ev al ob je ct ) ))
( se t f st ri n g
( conc at ena te ’ st r i ng
( subseq s t r i n g 0 in dex )
(when ( st ri ng p n ew− st rin g) new −st rin g)
(when (< new −ind ex le ng th )
( subseq s t r i n g new −ind ex ) ))
le ng th ( len gt h s t ri n g )) )
( ge t−t ex −l ine te x− s tr in g )
( get −tex −cha r te x− s tr in g lo oku p )) )
( t
( pro g1 ( ar e f s t ri n g i nd ex )
( unle ss l ook up
( in c f i nd ex ) ) ) ) ) ))
( t
( pro g1 ( ar e f s t ri n g i nd ex )
( unle ss l ook up
( in c f i nd ex ) ) ) ) ) ) ) ) )
( defun p ro ce ss− t ex −s tr ing ( te x− s tr in g )
"Process TEX-STRING."
( wi th −s lo ts ( st at e ) t ex −s tr in g
( loop : f or cha r := ( get −t ex− ch ar t ex −s tr in g )
: wh il e cha r
: i f ( eq char : e ol )
: do ( g et− t ex −li ne te x− s tr in g ) : and : do (s e t f st at e : N)
: el se
: do ( l e t ( ( ca tc od e ( cat cod e cha r ) ) )
( cond (( = c atc od e +esc ape +)
( l e t∗(( c har (g et− te x−c har te x− st ri ng ) )
( cat cod e ( cat cod e char )) )
( cond ( (eq ch ar : eo l )
( ea t ’ )
( se t f st at e :M ))
( (/ = c atc ode + l e tt e r +)
( ea t ( in t er n ( ma ke −s tr ing 1
: i ni ti al −e l em e n t ch ar )) )
( se t f st at e ( i f (= ca tco de +spa ce+ ) : S :M) ) )
( t
( l e t ( ( name ( l i s t char )) )
( loop : f or cha r := ( get −t ex− ch ar t ex −s tr in g
: lo oku p)
: wh il e ( and char
( not ( eq char : eo l ) )
(= ( cat co de cha r )
+ le t t er +) )
: do ( ge t−te x−c ha r t ex −s tr in g )
: do ( push char name)
: f i n a ll y ( ea t
( in t er n
( coerc e
( nre ver se name)
’ st r in g ) )) )
( se t f st at e : S) ) ) ) ) )
( (member ca tc ode ‘ ( ,+ be ginn ing −of− gr oup +
,+ end−o f−grou p+
,+ ma th −s hi ft +
,+ al ign men t−t ab+
,+ pa ra met er +
,+ s up er sc ri pt +
,+ s ub sc ri pt +
,+ l e tt e r +
,+ o th er +
,+ a ct iv e +)
: t es t # ’ =)
( ea t char c atc ode )
( se t f st at e :M ))
( (= ca tc od e +end −of −l ine +)
( sk ip −t ex −li ne te x− s tr in g )
( case s ta te
( :N ( ea t ’par ) )
( :M (e at # \S pace +sp ace + ))
( :S ) ))
( (= ca tc od e +ig nor ed + ))
( (= ca tc od e +spac e+ )
(when (e q st at e :M)
( ea t #\ Space +sp ace +)
( se t f st at e : S) ) )
( (= ca tc od e +com ment+ )
( sk ip −t ex −li ne te x− s tr in g ) )
( (= ca tc od e +i nv a l id + )
( warn "Invalid character ’~C’ in input." char ) ) ) ) ) ) )
( defun p ro ces s− st ri ng ( s t ri n g )
"Process STRING."
( pr oce ss −t ex− s tr ing ( te x− s tr in g st r i ng ) ))
18/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
Expected problems
Let’s be realistic. . .
Huge task
CFFI
Compatibility mode
T
E
X’s digestive engine is not really a pipeline
Lisp / traditional T
E
X interaction tricky
Sandboxing
Too much intercession. . .
All the things I haven’t thought of yet (a lot)
19/21
Star T
E
X
Didier Verna
Introduction
Why?
Common Lisp
Built-in paradigms
Extensibility
How?
API
Compatibility
Conclusion
These were the voyages,
Of a software enterprise.
Its continuing mission:
To explore new tokens,
To seek out a new life,
New forms of implementation.
To \textbf{go},
Where no T
E
X has gone before!
20/21
