Dyalog 09 Technical Road Map 09

Dyalog’ 09

Technical Road Map ’ 09 Morten Kromberg Dyalog’ 09 – Princeton, NJ

Agenda • Introducing Version 12. 1 – Demonstration of some Key Features • The Next Release Cycle – Version 13. 0 – Conga 2. 1, Encryption & Encoding Library • “APL#” - A new APL interpreter to complement Dyalog APL • Cirrus or Cumulo-Nimbus? • Eagle’s Eye View of Curves in the Road – Recommendations on preparing for the future Technical Road Map Dyalog’ 09 - Princeton 3

Version 12. 1 – Overview • It was supposed to be a “Year of Consolidation” • Let the new QA manager take charge • Build, Install & QA on every platform, every night • Find and Kill more bugs, Speed things up a bit • . . . But we managed to sneak a few new things into v 12. 1 Technical Road Map Dyalog’ 09 - Princeton 4

Version 12. 1 – Overview • • • Faster on Single Core, Some Multi-Core Features Editor Enhancements Fast XML-to-Array Conversion Tool (⎕ XML) Microsoft. Net components embeddable in Dyalog GUI User Command Processor for “Tool Delivery” IPv 6 Support with Conga v 2. 0 Journaling Component Files: Protection against O/S Crashes New Developer Tools: SAWS, Mild. Server, Load. Data, GRAPL Applications deployable as single. exe file (interpreter+ws) Improved Keyboard and Terminal Support I-Beam (⌶ ) allows experimental & undocumented features Technical Road Map Dyalog’ 09 - Princeton 5

V 12. 1 Core Performance v 10. 1 ms v 12. 1 ms Factor Boolean Scans and Reductions Highly Optimized mat← 1000 rand 2 +⌿mat vec← 1000000 rand 2 ≠vec “Set Functions” (⍳ ∊ ∪ 5200 51000 680 830 7, 6 61, 4 ∩ ~) Optimized for “One Small Argument” Transpose Rewritten vec~5 33000 6500 (vec≠ 5)/vec vec← 1000000 rand 100 11000 7200 5, 1 1, 5 ⍉ mat 32000 3 23000 ~0 1, 4 11, 3 4125 129 31, 9 23000 59000 240 230 95, 8 256, 5 ⍉ vec Indexing Uses Less Memory i j← 1000? ¨ 1000 mat[i; ] ⍝ (kb used) Indexing is also (often) Faster data←mat[i; ] mat[j; ]←data Technical Road Map Dyalog’ 09 - Princeton 6

V 12. 1 Core Performance v 10. 1 ms v 12. 1 ms Factor 500000 180000 2, 8 460000 480000 85000 34000 6300 16000 13, 5 76, 2 5, 3 8 2 3, 7 Grade Uses New (lexicographic) Algorithm ⍋ vec← 1 e 6 rand 1 e 9 For “Small Range” Data, Even Faster ⍋ vec← 1 e 6 rand 10000 “Sort” is now an idiom {⍵ [⍋⍵ ]}vec mat← 100000 4 rand 10000 {⍵ [⍋⍵ ; ]}mat Basic arithmetic (+ - ×) has been fully rewritten in assembly: Simple Arith uses “SSE” instructions where possible a b← 1000∘ rand¨ 50 ⍝ 1 -byte ints a+b More Optimized Cross-Type Operations a b← 1 0. 001×(1000)∘ rand¨ 50 1000000 ⍝ a: 1 -byte ints, b: floats a×b 16000 9000 1, 8 16000 7300 2, 2 Matrix iota idiom extended to numeric arrays data← 10000 10 rand 1 E 6 Technical Road Map data{(↓⍺)⍳ }data ↓⍵ Dyalog’ 09 - Princeton 7

Multi-Core Support • Most Scalar Dyadic Functions on Floating-Point arguments will use multiple cores (configurable) • But not + and -: Function 2 cores 4 cores + 12% - 12% × 12% ÷ 45 -50% 70 -75% GCD/LCM 45 -50% 70 -75% Technical Road Map Dyalog’ 09 - Princeton 8

Parallel Each Operator • A user-defined operator called PEACH is included with v 12. 1. It uses Conga to orchestrate multiple cores. • [Demo] • Parallel Each on a single machine works well with algorithms that have a good CPU: Memory ratio • “Distributed Each” works for any kind of calculation if transfer or arguments and results not too expensive Technical Road Map Dyalog’ 09 - Princeton 9

Performance Case Study • Visited company in Bruxelles doing “actuarial calculations” • V 12. 1 gave a 20% speedup “out of the box” • We found 20% more by doing “hot spot” analysis for a couple of hours • An 8 core machine (2 x. Quad) gave a 5 x speedup using “PEACH” on the inner loop of the application • Summary: 7 -8 x faster. 1 Hour calculation takes <10 minutes. Customer is pleased . Technical Road Map Dyalog’ 09 - Princeton 10

User Command Processor • Same idea as in APL*PLUS, but based on Unicode text files as source for user commands • A “User command” consists of a single Unicode Text file, all you need to do is drop it in a particular folder (no need to restart APL) • Makes it *VERY* easy to share development tools ]display 2 2↑ 3 3⍴'UCMDS' 'Rock!' ┌→────────┐ ↓ ┌→────┐ │ │ │UCMDS│ │Rock!│ │ │ └─────┘ │ │ ┌→────┐ │ │ │Rock!│ │UCMDS│ │ │ └─────┘ │ └∊────────┘ Technical Road Map Dyalog’ 09 - Princeton 11

IDE Enhancements • The Editor has been completely rewritten to better support editing large classes and functions • Outlining, Collapsible Regions, “Refactoring” • John Daintree will show you all the new features after lunch! Technical Road Map Dyalog’ 09 - Princeton 12

Fast XML Transformations • Converts XML text to (&from) an array: [; 1] Level of nesting [; 2] Element Name [; 3] Element Data [; 4] Attributes [; 5] Node type (0=Empty Tag, 1=Text only, 3=Has Children, etc…) • [; 4] not required when generating XML, [; 5] ignored Technical Road Map Dyalog’ 09 - Princeton 13

Integrated MS. Net GUI Technical Road Map Dyalog APL 14

Version 12. 1 – Overview • • • Faster on Single Core, Some Multi-Core Features Editor Enhancements Fast XML-to-Array Conversion Tool (⎕ XML) Microsoft. Net components embeddable in Dyalog GUI User Command Processor for “Tool Delivery” IPv 6 Support with Conga v 2. 0 Journaling Component Files: Protection against O/S Crashes New Developer Tools: SAWS, Mild. Server, Load. Data, GRAPL Applications deployable as single. exe file (interpreter+ws) Improved Keyboard and Terminal Support I-Beam (⌶ ) allows experimental & undocumented features Technical Road Map Dyalog’ 09 - Princeton 15

v 12. 1 Timeframe • Complete except for packaging and a rapidly diminishing number of bugs in the new editor • Target for General Availability on Windows+AIX is August/September 2009 (was “before” Dyalog’ 09) • Linux and PDA before year end • It isn’t too late to join the Beta-Test programme: Write to mkrom@dyalog. com and explain what you would like to test… • Monitor http: //www. dyalog. com or twitter. com/dyalogapl for notifications Technical Road Map Dyalog’ 09 - Princeton 16

Status – Spring’ 09 Dyalog APL 17

The Next Release Cycle • Conga 2. 1 • Version 13. 0 Technical Road Map Dyalog’ 09 - Princeton 18

Conga 2. 1 • Support for More Certificate Types: – We’ve had help from Peter-Michael Hager – Encrypted Certificates – The Windows Certificate Store (allows use of Smart. Cards and other “Tokens” that interface with this) • General available early in 2010 Technical Road Map Dyalog’ 09 - Princeton 19

Likely Features for ”v 13” • More Performance and QA work – Continued focus on multi-core, cache optimization and other aspects of modern chip architectures – “Industrial Strength” local and distributed PEACH • Remove “ 2 billion element” limit from 64 -bit version • New Data Types: – 128 -bit Decimal Floating Point (IEEE-754 -2008) for banking applications – Complex Numbers to attract educational users – Rational Numbers / Unlimited Precision Integers can wait – We have decided we do NOT NEED 64 -bit Integers Technical Road Map Dyalog’ 09 - Princeton 20

Likely Features for ”v 13” • Improve and rationalise internal developer tools: • ⎕ DMX: Diagnostic Message – e. Xtended – Provide more detailed information about errors • ⎕ MONITOR: Measure CPU and Elapsed Time with higher precision, easier to use, handle recursion • ⎕ STACK, ⎕ NSI to return “refs” • ⎕ NC, ⎕ NL… Rationalise “Metadata” or “Reflection” tools: Technical Road Map Dyalog’ 09 - Princeton 21

More Likely v 13 Features • Support for Unicode source files built-in to interpreter – Load, Save and direct Edit of Unicode Text files • General-purpose Encryption Library (Hashing, Symmetrical and Asymmetrical encryption) • Target availability for v 13 in “roughly 12 months” Technical Road Map Dyalog’ 09 - Princeton 22

Other Potential Projects • SQAPL Server: Allow an APL application to appear to be a Relational Database Engine (with ODBC, ADO. NET and JDBC drivers) – (This will not be a free product due to 3 rd party licensing) • Remote IDE for Dyalog. APL on all platforms • “Mono” Bridge – use. Net Framework under Linux • Provide More Developer Tools – as part of the APL Library. Cultivate an “Open Source” code repository, Technical Road Map Dyalog’ 09 - Princeton 23

”APL #” • We have started a research project, to produce a completely new, ”Fully Managed” APL Interpreter • We will publish a specification and discuss it, first with major customers and then ”in public” • Will be fully integrated with Microsoft. Net – but the core language syntax will be independent of. Net Technical Road Map Dyalog’ 09 - Princeton 24

APL # - Benefits • Applications written in APL# will be deployable in a ”secure” fashion over the internet, and in other scenarios where ”managed code” is a requirement or provides a significant advantage • For example, you will be able to write web applications with APL code executing in the browser to provide a ”rich client” experience Technical Road Map Dyalog’ 09 - Princeton L 25

APL # - ”Likely Characteristics” • Full set of APL primitives, operators and other core language features • Designed as a tool for APL thinkers – but comfortable for software engineers • ONE BIG difference: No ”semiglobals” (a function cannot see the locals of its calling environment) – There WILL be true globals – And the equivalent of namespaces will provide a ”reasonable” migration path Technical Road Map Dyalog’ 09 - Princeton 26

APL # - ”Likely Characteristics” • WILL PROBABLY NOT HAVE: – Auxiliary Processors, ⎕ ML, ⎕ OR, ⎕ SM, ⎕ RTL, ⎕ SD, ⎕ SR, ⎕ NTIE, ⎕ ARBIN, ⎕ W*, ⎕ AI, ⎕ SVO, ⎕ DQ, ⎕ D. . – (Library functions for which a perfectly good equivalent exists in the. Net framework) – Saved workspaces will not be able to have an )SI stack • First versions may not be able )SAVE workspaces at all • WILL PROBABLY HAVE: – ⎕ CT, ⎕ IO (jury is still out), and a small set of carefully selected library functions, Component Files, ⎕ XML (etc) which provide unique value to APL programmers Technical Road Map Dyalog’ 09 - Princeton 27

”Dyalog” vs ”APL #” • APL# is a complimentary product which allows the use of APL in places where it can not be deployed today • It is useful to organizations who need managed code, or use Visual Studio / C# or are otherwise heavily invested in Microsoft. Net • The rate of development of the core product will continue to accelerate • The two products will inter-operate seamlessly Technical Road Map Dyalog’ 09 - Princeton 28

Dyalog → APL # ”Interop” • APL# assemblies can be called by Dyalog APL (as any other. NET assembly) • Dyalog and APL# will (eventually) be able to share Component Files and communicate (many) APL arrays via TCP/IP – But not saved workspaces • Selected New Language Features in APL# will be ported to Dyalog APL – Will allow the writing of code which can be used in both environments Technical Road Map Dyalog’ 09 - Princeton 29

Timelines • APL# is still being designed, but see C: aplsharpsilverlightaplsharp. html • We will start reviewing the design externally this winter (come to APL 2010 to heckle) • A prototype may be available at APL’ 2010 – earliest ”General Availability” in 2011. • Pressure (and funding ) from customers could make us more ambitious about the features, but probably not the timing, of the first release Technical Road Map Dyalog’ 09 - Princeton 30

Cirrus or Cumulo-Nimbus? • A good home for ”Vapourware”? • Dave Thomas is about to teach us how to recognize and use different forms of clouds… Technical Road Map Dyalog’ 09 - Princeton 31

Amazon Web Services (AWS) • A ”mature” provider of ”web and computing infrastructure” (since 2006) • You provide a virtual machine image and Amazon ”scales it infinitely” at reasonable cost • We are investigating providing tools to support EC 2 (Elastic Compute Cloud), S 3 (Simple Storage Service) • Goal is to provide recipes, or pre-configured ”Amazon Machine Images” to build upon • We have a lot to learn, but hope to wrap our new knowledge up nicely for Dyalog users • No timeline. . . Talk to us if you have a potential app. Technical Road Map Dyalog’ 09 - Princeton 32

Thoughts about the Road Ahead Technical Road Map Dyalog’ 09 - Princeton 33

Looking Back Very Briefly Version 11. 0 (Autumn 2006) • Object Orientation – Classes – New Tool of Thought – Significantly Improved Integration with. Net – Source code management using standard tools • 64 -bit Version - LARGE Workspaces • Core Language Extensions – A number of new primitives: ⍣ ⌷ ∧ Technical Road Map Dyalog’ 09 - Princeton ∨ 34

Looking Back: v 12. 0 First Release (Win 32) in February 2008 • Main Features: – Unicode Support – Journaling Component Files – Simplified, Secure Communications Toolkit (“Conga”) – Built-In Charting and Document Creation – Language Bar and On-Screen Keyboard Technical Road Map Dyalog’ 09 - Princeton 35

Looking ”Back”: v 12. 1 October 2009, Key Features • Faster, more Robust • Multi-Core Support • Enhanced Editor • XML, Web Services • Tutorials and Tools – and more to come Technical Road Map Dyalog’ 09 - Princeton 36

To Come. . . • • • New “Commercial” and “Technical” Data Types More Speed – Especially On Multiple Cores Encryption and Other Web Enabling Tools Managed Code Interpreter More Ways to Run APL as a Service across Multiple Platforms (ODBC Data Source etc) Technical Road Map Dyalog’ 09 - Princeton 37

Recommendations • 64 -bit is Becoming Common, even on Windows – Good news: Porting to 64 -bit requires few code changes (once all the other components you need are available in 64 -bit) • Unicode – Please investigate what it will take to migrate – No hurry – but eventually, we will start adding new features in Unicode only (we’ll need more symbols ) • Multi (at least 2)-core machines are everywhere – Look at PEACH and other forms of ”multicore” splits Technical Road Map Dyalog’ 09 - Princeton 38

Recommendations • To prepare for the Web or Cloud - think about addding: – Web Front Ends using ASP. NET, Mild. Server etc – Web Services using ASP. NET or SAWS – An ODBC or ADO frontend using the SQAPL Server? • Take a look at Source Code in Unicode Text Files – Easier sharing and management – Access to many excellent (often free) tools! Technical Road Map Dyalog’ 09 - Princeton 39

Other Talks by Devt. Team. . . Today (Monday) 14: 15 John Daintree: The Version 12. 1 IDE 16: 15 Richard Smith: ⎕ XML, Journaled Files Phase II 16: 15 Roger Hui: Complex Numbers Implementation Stories 19: 30 John Scholes: Complex Numbers Tomorrow (Tuesday) 09: 00 Morten Kromberg: An ODBC Driver for APL 09: 00 John Scholes: Session Whizbangs Wednesday (Courses) 14: 00 Dan Baronet: Source Code Mgt using SALT and Sub. Version 14: 00 John Daintree: Using the Microsoft. Net framework 14: 00 Morten Kromberg: Multicore / Application Tuning Workshop Technical Road Map Dyalog’ 09 - Princeton 40

Conclusion • We’ve been keeping busy! • We’re starting to enjoy the benefits of being a significantly larger team. • We hope that you will enjoy some of the new Features - and Products - that are heading your way! • Thank You For Listening! Technical Road Map Dyalog’ 09 - Princeton 41

Other Talks by Devt. Team. . . Today (Monday) 14: 15 John Daintree: The Version 12. 1 IDE 16: 15 Richard Smith: ⎕ XML, Journaled Files Phase II 16: 15 Roger Hui: Complex Numbers Implementation Stories 19: 30 John Scholes: Complex Numbers Tomorrow (Tuesday) 09: 00 Morten Kromberg: An ODBC Driver for APL 09: 00 John Scholes: Session Whizbangs Wednesday (Courses) 14: 00 Dan Baronet: Source Code Mgt using SALT and Sub. Version 14: 00 John Daintree: Using the Microsoft. Net framework 14: 00 Morten Kromberg: Multicore / Application Tuning Workshop Technical Road Map Dyalog’ 09 - Princeton 42