Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Parallelizing Quickref

e ELS 2019 E

Didier Verna

EPITA / LRDE

didier@lrde.epita.fr

lrde/~didier

@didierverna didier.verna

in/didierverna

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Conclusion

Quickref parallelized

Performance improvement: 4x

Thank you! Any questions?

Parallelizing Quickref / ELS 2019 – Didier Verna 2/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Introduction

Reference manuals for CL

(Quicklisp) libraries

quickref.common-lisp.net

or local builds

Originally: sequential loop

over 1700+ libraries

1h30 – 7h

depending on the conditions

Parallelism worth investigating

Parallelizing Quickref / ELS 2019 – Didier Verna 3/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 4/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 5/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Toolchain

Quicklisp

foo/

Declt

foo.texi Makeinfo foo.html

Important remarks:

Declt works by introspection

Compilation / loading (of dependencies) may be required

Avoid loading 1700+ libraries in the same Lisp image!

Run Declt in external processes

Makeinfo is a Perl/C script

Ditto

Parallelizing Quickref / ELS 2019 – Didier Verna 6/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 7/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Scenarios

Quickref Options

Libraries and Update

full / installed-only

download / compilation may occur

Compilation cache Policy

global / local

global cache may cause problems

Scenarios

1. All libraries already compiled (1h 27m)

2. Nothing compiled, global compilation cache (1h 51m)

3. Nothing compiled, local compilation cache (7h 01m)

Parallelizing Quickref / ELS 2019 – Didier Verna 8/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Time Distribution

Makeinfo

48!%

Declt

48!%

4!%

Makeinfo

40!%

Declt

41!%

3!%

Compilation

16!%

Makeinfo

10!%

Declt

10!%

1!%

Compilation

79!%

Scenario Texinfo Html

1 52% 48%

2 60% 40%

3 90% 10%

Parallelizing Quickref / ELS 2019 – Didier Verna 9/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 10/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Parallel Solution

Library Pool

Declt

Texinfo Files Makeinfo

Makeinfo

HTML Files

Usable in all scenarios

Dependency graph management cheap

Scenario 2

Best results: 4 Declt threads / 4 Makeinfo threads

Total time: 28m 17s (25% of sequential time)

Parallelizing Quickref / ELS 2019 – Didier Verna 11/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Library Pool Management

Dependency Graph

1 2

4 3

Library Pool Done

2 1

3 4

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Library Pool Management

Dependency Graph

1 2

4 3

Library Pool

1 2

Done

2 1

3 4

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Library Pool Management

Dependency Graph

1 2

4 3

Library Pool

Done

2 1

3 4

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Library Pool Management

Dependency Graph

1 2

4 3

Library Pool

Done

2 1

3 4

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Library Pool Management

Dependency Graph

1 2

4 3

Library Pool

Done

2 1

3 4

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Library Pool Management

Dependency Graph

1 2

4 3

Library Pool

Done

2 1

3 4

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Library Pool Management

Dependency Graph

1 2

4 3

Library Pool Done

2 1

3 4

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 14/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Discussion and Perspectives

Alternative Algorithms

Dependency Management Issues

Based on static information provided by Quicklisp

Fragile, not always correct

CPU vs. I/O Consumption

4x is a bit disappointing

Open the Declt and Makeinfo black boxes

SJF-like Scheduling

Very dicult to gure out where complexity comes from

Collect timings and use them in next run

SSD!

Parallelizing Quickref / ELS 2019 – Didier Verna 15/16

Conclusion

Introduction Toolchain

Experiments

Solution

Perspectives

Acknowledgments

Initial code base: Antoine Martin

Author index and parallel algorithm #4: Antoine Hacquard

Hosting (code & website): CLF / common-lisp.net

Parallelizing Quickref / ELS 2019 – Didier Verna 16/16