[Mythryl] Status update

cynbe at mythryl.org cynbe at mythryl.org
Sun Oct 18 20:52:42 PDT 2015


First, personal:  I figured I'd be pretty much home free after my
primary colorectal tumor was removed (about 66% chance) but I wound
up with metastatic tumors throughout my torso.  At last check, the
biggest one was growing smaller and all the smaller ones were growing
bigger.

My oncologist says I have 2-3 years to live, maybe 4, barring medical
breakthroughs.  (On the other hand there is a LOT of breakthrough-level
activity in the cancer-treatment field this year centering on immune-based
treatments, so maybe I'll live long enough to benefit from that. Five-year
survival statistics are always necessarily retrospective, reflecting the
past state of the art in treatment rather than the current or near-future
state of the art.)

I've spent a decade developing Mythryl in the expectation of 2-3 decades
of return on investment, but now it looks like I cannot count on more
than 2-3 years of return on investment, so I guess the joke is on me.

It is also worth noting that chemo fatigue is cutting heavily into my
Mythryl programming output -- I just came off 10 days of no Mythryl
commits, where before chemo I rarely went a day without a commit -- and
my productivity is likely to continue to decline as the dying process
proceeds, both from fatigue and from decline in motivation to work on
it as the return-on-investment period shrinks.

Judging from the activity level on this list when I don't post,
Mythryl will die with me.

That's a bit of a shame, because it means the mainstream world is
going to spend about 20-30 years re-inventing Mythryl when it could
have adopted it directly and saved a bunch of time.  Mythryl represents
about an eight-figure development effort in commercial terms[1], plus
it is vendor-neutral in an era when we're returning to vendor-specific
languages: Java is Oracle's property, F# is Microsoft's, Go is Google's
etc.  This will be hard on the open-source world.

(I think many open source programmers today don't remember the
pre-C/Unix world when every major vendor had its own programming
languages and ecology, with the major exceptions of Fortran and Cobol.
C/Unix was different because they came from Bell Labs at a time when
owner AT&T was a regulated monopoly forbidden from entering
commercially into software and operating systems, so it released
C+Unix "free" (about a $100 tape copying charge, iirc) which let it
take over first the academic world (resulting in BSD Unix) and then
the commercial world -- the 1980s "Unix wars", which ended with the
death of countless commercial competitors to Unix. But now both AT&T
and Bell Labs are dead (the current "AT&T" is just a renamed Southwest
Bell) and there won't be any more gifts like that to the software
world, so we'll have to either make our own via the Linux Foundation
or such, or else drift back into a fragmented world of per-vendor
software ecologies with little in the way of portability between them
(which seems to be the trend at present).

Anyhow, that won't be my problem, I'll be pushing up daises shortly,
according to the current oncology odds-makers.

But anyone thinking of keeping Mythryl alive beyond 2-3 years out might
want to think about learning how to maintain the codebase while I'm
still around to answer questions.  Fair warning.




Second, Mythryl.  I see the last release date is 2013-04-13 -- about
two and a half years ago.  There aren't likely to be any more gaps
that long between releases from me, since that is about my current
life expectancy. :-)

    A bit of nomenclature before I proceed: I refer to server microthreads
    as "imp"s.  Think of an imp as a small daemon.  Similar idea, but a
    daemon is a complete Unix process in its own address space, while an
    imp is one of many microthreads sharing the Mythryl process address
    space.  (What distinguishes an imp from a random microthread is that
    an imp supports Mythryl's protocol for imp startup, shutdown, and
    intercommunication.)  Imps are nice in that they can pass around
    complex datastructures with a single pointer (due to the shared
    address space) and complete typesafety, whereas daemons can communicate
    only painfully by value via untyped bytestreams offering basically zero
    typesafety.

For the upcoming release I have done a few final tweaks to the Mythryl
language/compiler itself; at this point there are no irritating-to-me
errors of commission in the language to speak of.  Changes in this
release include changing '\010' escapes in strings and chars to be
octal as the mainstream expects instead of decimal as the ML creators
decided for some reason, and freeing up a few more reserved words for
general use, for example replacing 'fn' by '\\' so as to be able to
use 'fn' as a parameter name for an arbitrary function ("closure" if
you're picky) getting passed in as an argument -- everyone wants to
use 'fn' as the name. 
    (About the only thing that bugs me is that one needs to use '==' to
replicate a datatype but can use '=' to replicate any other type.  I'd
be very happy if someone would fix that -- I gave up after convincing
myself that it would require major restructuring of the front end that
I was simply not prepared to spend the required amount of time to
accomplish. Sometimes ten years of infrastructure work is enough.)

But mostly it has taken awhile because this upcoming release is
basically about converting Mythryl from a nice compiler for a nice
language into a viable software platform for GUI application
development.  I'm very interested in having a system as
programmer-friendly as the old Lisp Machine and Forth environments,
where one could hack up quick solutions interactively in seconds, instead
of hours or days of batch-mode edit-compile-run-debug.

This has involved:

1)  Reworking John H Reppy's CML ("Concurrent ML") a bit and
    working out a viable set of design patterns for concurrent ML
    programs.  This went fairly slowly because I was in the position
    of breaking new trail almost all the way, and that goes a lot
    more slowly than following other people's blazed trails.
    The Mythryl CML-ish concurrency support centers on
       src/lib/src/lib/thread-kit
    -- about 12,000 lines of code.

2)  Reworking John H Reppy + Emden Gansner's original CML X client
    lib code that puts X packets on the wire via a socket and then
    converts the replies back into Mythryl datastructures.  This
    was relatively easy;  John + Emden wrought really well in this layer.
    The main work was retrofitting improved design patterns.
    The Mythryl x-client library centers on
       src/lib/x-kit/xclient/src/wire/
    -- about 12,000 lines of Mythryl code.

3)  Reworking from scratch John H Reppy + Emden Gansner's original CML
    X widgets. They basically tell you in their docs that you'll have
    to do this, and they were right.  (Remember that this was all part
    of John H Reppy's PhD thesis which centered on the design of CML;
    implementing CML was an extra and implementing the CML X client
    library was an extra on an extra and implementing the CML X client
    widgets was an extra on an extra on an extra.  If you've done a
    PhD you know how intense the time crunch is to get everything
    submitted and get your sheepskin.  So them running out of time
    to do all the CML X widgets up proud is extremely understandable.
    It is astonishing how much stuff they managed to do, not how much
    stuff was left to do.)
    The (all-new) X widget layer centers on
       src/lib/x-kit/widget/leaf/
    -- about 18,000 lines of Mythryl code.

4)  Designing and implementing from scratch a guiboss-imp.pkg to
    provide centralized services for running GUI apps.
    The guiboss-imp layer centers on
       src/lib/x-kit/widget/gui/guiboss-imp.pkg
    -- about 16,000 lines of Mythryl code.

5)  Designing and implementing from scratch a coarse-dataflow
    application framework together with emacs-flavored editing
    support based on that framework.
    The coarse-dataflow application framework
    + emacsish functionality centers on
       src/lib/x-kit/widget/edit
    -- about 28,000 lines of Mythryl code.       


The above total about 86,000 lines of Mythryl code, so two and a
half years of spare time to complete the work is not really all that
unreasonable.  Any of the above would qualify as an open-source
project in its own right, if split off.





Expanding on the above in a bit more detail:

1) CML tweaks and design patterns.

The orignal CML concurrency support centered on the use of
blocking rendezvous (no information transfer until both
microthreads are ready) on the grounds that this maximizes
coupling between the microthreads.  (In retrospect, it seems
clear that in a modular software design one wants in general
to minimize rather than maximize coupling between different
software components.)

The original 1990 CML concurrent-widgets design also centered
on trees of widgets communicating exclusively bidirectionally
parent-to-child, with separate event streams for keyboard and
mouse events.

The original CML concurrent GUI apps design was also completely
decentralized.

Lessons from working with this system:

 *  Separate keyboard and mouse streams is bad because ordering
    between events on the two streams is lost and almost impossible
    to recover.

 *  Bidirectional parent-child communication together with
    blocking rendezvous communications results in constant
    deadlock bugs.

 *  Total decentralization is a great PhD project learning
    experiment, but not very practical for production use:
    Often you need to see the forest, not just a tree or two.

 *  You need a multiphase protocol for starting up graphs
    of concurrently communicating microthreads, so that
    you can reliably
      o  Create the graph nodes (microthreads).
      o  Add the graph links (tell the microthread about their graph neighbors).
      o  Start all the microthreads running (only) after everything is set up.

 *  You need some systematic design pattern for avoiding deadlocks
    and race conditions -- it isn't enough to just give the application
    programmer a few low-level concurrency primitives and say
    "Good luck, kid!" and wash your hands of the issue.


Learning from those who went before, I have in response
to the above chosen to base the Mythryl GUI support on:

 *  A centralized guiboss-imp.pkg which provides global services.

 *  A single input event queue per imp.

 *  Mail queues rather than mailslots for imp input.  This lets
    clients send to an imp with blocking until the imp is ready.

 *  A three-phase impgraph startup protocol
     1) Create the imp 'eggs' and get their exported interfaces.
     2) Pass to the eggs the imported interfaces they require.
     3) Fire a starting gun which lets them all begin operation.
    For a working example of this look at  make_windowsystem_egg()
    in  src/lib/x-kit/widget/xkit/app/guishim-imp-for-x.pkg

The above still does not deal sufficiently with the issue of deadlock avoidance.

Looking at actual examples, deadlock is typically due to several communicating
imps which have multiple blocking input statements in their event loop,
allowing situations to arise where each is blocked waiting for another to
respond, and consequently each never progresses to the input statement which
would allow it to respond.

Scratching my head for a few years, I concluded that the clean solution
is to adopt a design pattern of:

    EXACTLY ONE BLOCKING I/O STATEMENT PER IMP EVENT LOOP.

Achieving this requires some work, because often an imp will issue a request
to another imp and want to continue with the relevant information when the
reply arrives.  It is very hard to avoid this using the original CML primitives,
so what I did was:

 *  Introduce a global Replyqueue type, and equip each imp with its own
    replyqueue named 'to'.

 *  Introduce a new   do_one_mailop'  concurrency prim which is just like
    the existing      do_one_mailop   concurrency prim except that it takes
    an additional Replyqueue argument, and will read messages from that
    replyqueue as well as the other specified sources.

 *  Use the new   do_one_mailop'  prim as the sole blocking input prim
    in each imp.
    Working example from     src/lib/x-kit/widget/xkit/app/guishim-imp-for-x.pkg:
        #
        fun loop ()
            =
            {   do_one_mailop' to [
                                     (end_gun' ==>  shut_down_appwindow_imp'),
                                     (take_from_mailqueue' appwindow_q    ==>    do_appwindow_plea)
                                  ];

                loop ();
            }
                                             
Here  end_gun'  provides a global way to shut down an entire impgraph simultaneously
(another important imp protocol element in practice) and take_from_mailqueue' handles
requests from other clients while while the 'to' arg provides the Replyqueue needed
to handle replies from other imps.  In the body of the imp one then writes calls like

    cpt::pass_clientside_pixmat_from_window
       (window_rectangle_to_read, window)
       to
       {. handle_pixmat #pixmat; }
       ;

Here 'to' is the local imp's Replyque and
    {. handle_pixmat #pixmat; }
is the continuation which  do_one_mailop'
will execute when the reply to the
    cpt::pass_clientside_pixmat_from_window
call arrives in due course;  in the meantime
the local imp proceeds with normal event loop
processing, remaining responsive to all incoming
requests.


Incidentally, as another little design pattern, the original 1990 Reppy/Gansner
code had each imp declare a datatype of all supported messages and then pass
instances of that datatype to the event loop.  This turns out to be
needlessly clumsy:  Instead one can have the input loop of just about
every imp look like

        fun loop ()   # Outer loop for the imp.
            =
            {
                do_one_mailop' to [
                                    (take_from_mailqueue' mailq ==>  do_plea)
                                  ];

                loop ();
            }
            where
                fun do_plea thunk
                    =
                    thunk runstate;
            end;

and then have individual messages to the imp structured like

    fun pass_window_site (replyqueue: Replyqueue)  (reply_handler: g2d::Window_Site -> Void)
        =
        {   reply_oneshot =   make_oneshot_maildrop():    Oneshot_Maildrop( g2d::Window_Site );
            #
            put_in_mailqueue (appwindow_q,
                #
                \\ ({ me, ... }: Runstate)
                    =
                    put_in_oneshot (reply_oneshot, site)
            );

            put_in_replyqueue (replyqueue, (get_from_oneshot' reply_oneshot) ==> reply_handler);
        };

This is much more flexible and maintainable than the datatype approach.



Anyhow, in practice I have not found much need for the above pass_foo()/Replyqueue
in coding written to date -- it is usually pretty clear from local
analysis that there is no deadlock danger -- but it provides a
comforting general design pattern solution to which I can retreat
if some subsystem starts developing deadlock issues.


Race conditions are a separate issue.  One helpful design pattern
is to make impgraphs communicate on a spanning tree, with only one
path between any given two imps.  Sometimes this means restructing
an imp to pass through information it is not interested in, just to
avoid having a bypass path for that information which introduces
possible race conditions due to multiple communication paths between
some pairs of imps.

In the GUI context, one may be starting up and shutting down parts
of a GUI;  I'm finding an uncomfortable number of opportunities for
race conditions in the startup phase where different parts of the
system are starting up "simultaneously" in contexts where the
above imp/egg startup protocol does not apply.  So far I haven't
found a nice design pattern to solve these issues in a general
way, so for the time being I've retreated to ad hoc solutions on
a case-by-case basis.  *sigh*


Another basic concurrent-program design pattern I've adopted is the
use of multiple interfaces per imp, in general.  I specify these in
very small packages like

    src/lib/x-kit/widget/theme/guiboss-to-guishim.pkg

where the name indicates the two parts of the system which are
expected to be communicating via this interface.  Having imps
refer to these little port-specifying packages rather than directly
to each other avoids a plague of cyclic-package-dependency problems.
   In some cases a group of these port definitions just really,
really want to refer to each other in cyclic ways.  I fight this
at first, but if they seem sufficiently determined I give in and
put them all in one big file where they can recursively refer to
each other to their heart's content.  I view these as datastructure
black holes;  Once a datastructure moves into such a file it is
almost impossible to extract it again.  Two leading examples:

    src/lib/x-kit/widget/gui/guiboss-types.pkg
    src/lib/x-kit/widget/edit/millboss-types.pkg



2)

The Mythryl x-client library worked very nicely without anything but
renaming and commenting.  It would be nice to add support for some
X extensions like DGL.  (DGL is a lot like OpenGL 1.0 but runs over
the X stream, hence is usable remotely, whereas OpenGL is typically
only usable on the local machine.)


3)

The (all-new) X widget layer work mostly centered upon establishing
the right framework and interfaces;  the widgets themselves represent
a week or so of work recreating Reppy+Gansners original widget functionality
in the new framework.  (The framework is so different that all I managed
to salvage from the old widgets was a few lines of draw code.)

The new X widget framework provides an interesting study in implementing
functionality normally implemented in "object-oriented" ways in a
nominally non-OOP mostly functional language.  After a fair amount of
experimentation, I wound up doing this primarily by providing emacs-style
"hook" functions at various levels to allow customization by overriding
selected functions in a given package/imp instance.  Depending on your
temperament, this is either a lot clumsier than OOP overriding, or else
a lot more structured than OOP overriding, providing precise definitions
of which functions can be overridden and precise interfaces for doing
so in various ways at various times in the imp lifecycle.

The prime example to look at here is

    src/lib/x-kit/widget/xkit/theme/widget/default/look/widget-imp.pkg

This is in essence the common superclass for all Mythryl widgets, with
the widgets in

    src/lib/x-kit/widget/leaf/

being concrete specializations.

This process can be repeated over multiple levels, for example

    src/lib/x-kit/widget/edit/textpane.pkg

is built on widget-imp.pkg but exports its own hook functions which
other clients can in turn specialize.

I don't know if this approach will win over many OOP addicts, but it
does seem to work quite cleanly and it has the advantage of not
needing mystery code buried in the compiler in order to work.



4)

The guiboss-imp layer provides global functionality to start up and
shut down GUIs.  This proved more involved than I expected:

 *  I wanted to decouple it from X, so as to be able to run the Mythryl
    widgets on OpenGL, on web browsers, and on simple framebuffers (for
    example) so I introduced an abstraction boundary missing on the
    original design, namely the above-mentioned
    
        src/lib/x-kit/widget/xkit/app/guishim-imp-for-x.pkg

    all the X-specific stuff lives below this shim, and all the
    widget stuff lives above the shim, insulated from direct
    X dependencies by the shim bidirectional translation layer.

 *  I wanted popups and scrollables.  This turned out to involve
    basically building a little window system in Mythryl, using
    fast X rectangle-copy operations to make it fast.

 *  The original design had a fair amount of built-in latency.
    The new design has virtually none except:

 *  I wanted to avoid having frequent widget updates thrash the
    rendering subsystem, so I introduced a two-stage protocol
    in which
    
      1) A widget which changes state notifies guiboss-imp that
         it needs to be redrawn.
         
      2) About 20 times a second, guiboss-imp notifies redraw-needing
         widgets that it is time to redraw themselves, at which point
         they send guiboss-imp a graphics expression of their current
         appearance, which gets forwarded on via the shim.
         
    This introduces a 50ms latency on widget updates, which is normally
    fine.  (Any widget really requiring more rapid updates is free to
    bypass the above protocol and just send a redraw immediately. But
    for most analytics/visualization purposes it is nice to have the
    GUI rendering layer throttled so as to leave most CPU time for the
    actual core computations.)

A particularly nice part of the current design and implementation is
the support for dynamic reconfiguration of parts of running GUIs.
This is based on three interacting datatypes plus a good deal of
support from guiboss-imp, and took quite a few cycles of exploratory
programming to get right.

We use three representations for a GUI with transition diagram

     Guiplan    # Public definition of GUI constructed by client and then passed to
         |      # guiboss_imp via Client_To_Guiboss.start_gui or Gadget_To_Guiboss.make_popup.
         v
     Guipane    # Primary representation of a running gui.  This is private to
       |   ^    # guiboss_imp and its private support packages.
       v   |
     Guipith    # Public summary of a running gui.  Its purpose is to allow client
                # code to morph Guipane(s) by generating a Guipith
                # (Gadget_To_Guiboss.get_guipiths), editing it, and submitting it
                # (Gadget_To_Guiboss.install_updated_guipiths).

Guiplan lets client code specify a new GUI without being exposed to excessive detail.

Guipane is the guiboss-internal representation for a running GUI.

Guipith is a very schematic representation of a running GUI, which is
used by having a client request the guipith for the currently running
set of GUIs, editing it fairly arbitrarily including moving widgets
around in the tree, dropping widgets and adding new widgets via embedded
Guiplan fragments.  This is a very powerful technique for implementing
dynamic GUIs.  The basic technique can be seen in (for example)
    split_pane_vertically()
    delete_other_pane()
    rotate_panepair()
in
    src/lib/x-kit/widget/edit/fundamental-mode.pkg

If you look at the above code you'll note two little conveniences

  do_while + do_while_not
  
    I finally had a need for these loops so I implemented them in
    
        src/lib/core/init/pervasive.pkg
    as
        fun do_while (fn: Void -> Bool): Void
            =
            if (fn ())  do_while  fn;
            else        ();
            fi;

        fun do_while_not (fn: Void -> Bool): Void
            =
            if (fn ())  ();
            else        do_while_not fn;
            fi;

  gtj::guipith_map
    You'll find this implemented in
        src/lib/x-kit/widget/gui/guiboss-types-junk.pkg
    It demonstrates how to extend map/apply style functionality
    to large recursive datatypes, allowing client code to make
    little edits with minimal effort.  This is another major
    design pattern.




5)

The coarse-dataflow application framework provides the first real glimpse
of where I am ("was"?) going with all this.  It provides an almost-usable
implementation of emacs-style editing functionality (implemented completely
differently -- "buffer" contents are fully persistent and every line in
a textpane is animated by a separate concurrent microthread -- microthreads
only cost about 100 bytes in Mythryl, with no stack or such, so this is not
extravagant by any means) but it should not be thought of as a programming
editor or IDE so much as a framework for coarse-grain GUI application
development, inspired in part by emacs but more by similar coarse-grain
GUI application frameworks like AVS ("Advanced Visualization System" -- a
staple app in scientific visualization) or perhaps Labview or Linux's
gstreamer (which I'm unfamiliar with).

The basic idea is that one can start up a set of 'mill' microthreads which
do event-processing style computations from an input stream to an output
stream (in the most general case), and which can then be interactively
hooked together in coarse-grain dataflow networks to rapidly compose
custom analytics, display or synthesis applications.

Each 'mill' may be displayed in zero or more 'panes', which are in essence
mini-GUIs visible on the screen.  In the emacs-ish programming editor
context, each 'mill' corresponds loosely to an emacs 'buffer' and typically
holds one file being edited (but dired-mode holds a directory, eval-mode
has a buffer hosting interactive Mythryl code evaluation, shell-mode
controls a bash shell running in a subprocess etc), and each 'pane'
displays the state of one mill.

Following the emacs lead, each mill may have multiple panes open on it,
which may display different things, for example different parts of the
same file in the program editing context, or different parts of the
same stockmarket time-series or different 3-D views of the same 3D
scene.

There will shortly be an interactive pane for editing the graph of
running mills:  The code to date is in

    src/lib/x-kit/widget/edit/millgraph-mode.pkg

Interactive evaluation in an emacs-ish buffer is implemented and running
smoothly:

    src/lib/x-kit/widget/edit/eval-mode.pkg

That was one of the bigger recent projects and was a relief to complete,
since the SML/NJ codebase was never intended to support such functionality
(although the shell-level interactive evaluation mode code was a great
starting point).

I am deeply offended that computers, originally intended as
labor-saving machinery, have, thanks to modern GUI maldesign, turned
into labor-creating machinery. Consequently there are entire buildings
all over the globe full of people who do nothing but click on GUIs all
day long -- work which would easily be automatable in a commandline
context.

The Mythryl pane/mill framework is intended to allow use of GUIs without
falling into that trap:  My vision is that in general each mill will be
a specialization of the core emacs-ish program editor's

    src/lib/x-kit/widget/edit/eval-mode.pkg

with functionality augmented by a pane for graphics

    src/lib/x-kit/widget/edit/drawpane.pkg

For any given mill, then, one should in general be able to view a GUI
style display of its state; but also, at need, open an eval-mode style
interactive-Mythryl-code-evaluation subpane and interact with or
customize the mill in ways not envisioned or supported by the GUI
pane implementor for that mill.

There's an

    sh/mythryl-emacs

script in the current github codebase which allows experimenting with
the Mythryl pane/mill paradigm.  It comes up as a window on a little
test text buffer (incidentally demonstrating the system's ability to
display UTF8 -- all the textbuffer logic for the editor is written
to support UTF8, and I've made a few changes in string.pkg such as
replacing the old string::length with string::length_in_bytes +
string::length_in_chars) supporting most of the common emacs keystroke
definitions (see fundamental-mode.pkg for the keybindings and definitions),
but you can then do   M-x eval  to open an evaluation buffer
or   C-x f  to visit a file or such.

To get a sense for how the system is intended to allow rapid construction
of custom interfaces you should play around with the keybindings

   C-x }
   C-x 1
   C-x 2
   C-x 3
   C-x ^
   C-x o

As in emacs, "C-x 2" and "C-x 3" are the GUI equivalent of posix
fork(), replacing the current pane by two identical panes onto the
same mill as before.  You may then repurpose either of both of those
panes via (say) "C-x f" to open a different file (start a different
mill+pane pair), just as after fork() one may use exec() to replace
the contents of the current process.

The result of doing many "C-x 2" and "C-x 3" operations is a binary
tree of panes.  (This is important to understand and remember!)

The pane currently having the keyboard focus is clearly marked by a
big black surround.

You may use "C-x o" to move the keyboard focus around between panes.
(You may also use a mouseclick on the desired pane, but usually you
don't want to take your hand off the keyboard.)  As in emacs, doing
"C-x o" will cycle through all available panes.

Unlike emacs, in which multi-pane functionality is fairly lightly used,
in mythryl-emacs each pane is assigned a small-integer id, displayed
at the left of the modeline, allowing one to move the keyboard focus
directly to the desired pane via the "universal prefix" convention:
    C-u 13 C-x o
will move the keyboard focus directly to pane 13.

As in emacs "C-x 1" may be used to delete a pane.  Unlike emacs, in
which "C-x 1" removes all panes but the one with keyboard focus, in
mythryl-emacs "C-x 1" removes only the sibling of the current pane
in the pane binary tree.  Doing "C-x 1" repeatedly will eventually
reduce the display to just the current pane.  Again, this change in
semantics is because I envision using complex multipane displays much
more heavily than is typical in emacs:  The revised mythryl-emacs
semantics provides finer control over reconfiguration of the panetree.

As in emacs, "C-x ^" may be used to re-apportion screenspace between
the current pane and its sibling in the pane binary tree.  Unlike
emacs, in mythryl-emacs this operator works both horizontally and
vertically, and it is typically used with a numeric prefix giving
the percent of shared screenspace to be allocated to the current
pane.  E.g.  "C-u 10 C-x ^" will give the current pane 10% of the
space allocated to the panepair, with the remaining 90% going to
its sibling.

The mythryl-emacs "C-x }" op is totally unlike that in emacs.
The mythryl-emacs version rotates the current panepair 90 degrees:
   If the current pane is on the top,    it will wind up on the right.
   If the current pane is on the right,  it will wind up on the bottom.
   If the current pane is on the bottom, it will wind up on the left.
   If the current pane is on the left,   it will wind up on the top.

The above operations are all useful in the mythryl-emacs context of
editing multiple files, but they are intended more generally for
re/constructing dynamic graphics GUIs for analytics and visualization
purposes, once that functionality is complete and some appropriate
application-specific mode+mill pairs have been written to support
time-series analysis and stock trading or control of a CNC mill
(mine's a nice little Sherline) or such.



On a separate front, I've added some nice simple fully-persistent graph
implementations:

    src/lib/src/digraph.pkg
    src/lib/src/digraphxy.pkg
    src/lib/src/tuplebase.pkg

These still need some work to attain final form, but they provide great
general-purpose representations for the state of various systems.  Since
they are fully-persistent, creating new states is quite cheap, and a
metagraph with nodes having graph values and edges having graph-delta
values may be used to track the set of states explored to date together
with the transitions between them.  I intend to use this as a core bit
of infrastructure in computer vision, assuming I live long enough to get
to that.  I implemented all three because I'm not sure which one(s) will
prove most useful in practice.  Check out the matching .api files for
more information.

(I have also, after a couple of decades of mulling the problem over,
found a nice fully-persistent implementation for a triangle-mesh
boundary-reprentation of 3D solids supporting constructive solid
geometry operations.  That made me very happy. :-)  I don't know
when or if I'll find time to actually implement that.)



Other thoughtlets:

I recommend writing apps as little scripts which reference libraries;
mythryl-emacs may be taken as a model.  This avoids the startup-shutdown-startup
issues involved with building ersatz Mythryl "binary executables", which
are in many respects really an abomination.  Probably the mythryld image
should be the only such "executable".

As a small but important production-quality touch, at some point I
implemented support for a

    MYTHRYL_LIB_LOAD_PATH

environment variable with default value

    .:$HOME/.mythryl/lib:/usr/lib/mythryl:/usr/local/lib/mythryl

which lets you load libraries in a script without having to hardwire
the full path to the library.  The code for this may be found in

    src/app/makelib/main/lib-load-path.pkg

For an example of using this see the

    use "src/lib/x-kit/xkit.lib";

line in

    sh/mythryl-emacs

Notice that as above, you will typically want to give a short relative
path for the library.  This is because unlike the typical Linux practice
in which libfoo.a files are placed directly in /usr/lib or such, in
Mythryl we support use of both frozen and thawed library without any
change in the scripts using them.  This works by having the scripts
reference the

    foo.lib

file specifying how to build the library rather than the

    foo.lib.frozen

file actually containing the compiled code.  When Mythryl is asked
to "use" a given "this/that/foo.lib" it first checks for

    foo.lib.frozen

(even though it is not explicitly mentioned) and if it is present
uses it immediately (without even checking to see if foo.lib exists);
if foo.lib.frozen does not exist, Mythryl reads the "foo.lib" file,
checks the relative dates on all the relevant source and object files,
recompiles any out-of-date objectfiles, then loads them in directly
without constructing a foo.lib.frozen file at all.

This arrangement is useful because it allows skipping construction
of foo.lib.frozen during development, with each script invocation
automatically doing any required recompiles, and then for production
use freezing the library, allowing much faster startup (since the
dates on all the source and object files do not need to be checked).

But since .lib files generally need to live down somewhere in the
source directory hierarchy for a given application, and the .lib.frozen
file needs to live next to it (consider the case where an app has
multiple regex.lib libraries in different subdirectories -- the SML/NJ
codebase has something like a dozen separate regex libraries :-( ), we
really want to search for paths rather than assuming all libraries
are crudely dumped in a single lib/ directory somewhere.

Consequently, when one puts some library like xkit.lib.frozen in
/usr/lib or /usr/local/lib for general production use, one will in
general want to install it as

    /usr/lib/src/lib/x-kit/xkit.lib

not /usr/lib/xkit.lib

Got that?  Not complicated, but likely to be initially confusing to
people used to the simple traditional unix library search model.



A couple more Mythryl design patterns:

Just about any record exchanged between imps will probably at some
point need to be stored in an indexed structure, typically a red-black
tree.  In C you can use the record's address as a name for it, because
the C memory management model is basically "Nothing moves. Ever."  The
Mythryl memory management model by constrast involves moving data items
around in memory about 200 times per second (as a typical rule-of-thumb
number), so using item addresses is a really bad idea and in fact there
is no official supported way of getting such addresses.
    To help with this problem I've installed type Id and fn
issue_unique_id: Void -> Id in pervasive.pkg (making them available in
all packages) plus provided a id-map.pkg in src/lib/src parallel to the
existing string- and int-keyed redblack trees.  So I usually include an
    id: Id
field at the start of each widely-visible record type, and instantiate it as
    foo = { id => issue_unique_id(),
            ...
          };
If you adopt this habit, it will save you significant grief. :-)

The general pattern for an imp is usually to store a bunch of state in
(e.g.) redblack trees mapping Id -> My_Record for some My_Record, and
save the redblack trees in refcells.  If My_Record is pure (no direct
or indirect references to refcells or mutable vectors) then the entire
redblack tree can be passed out to clients without endangering the
integrity of local state, since nothing clients can do with the trees
they are given can affect our local state.  This is often a simple and
effective way of communicating complex state.  For example the Mythryl
emacs-ish textbuffer uses a simple
    nl::Numbered_List( Textline )
pure datastructure to hold the text in the buffer, which means it can
hand out copies of the entire buffer freely to clients without compromising
its control of its state.  (Numbered_List, btw, is a nice datastructure
which deserves to be more widely known and used.  And using it means that
emacs-style "undo" can be implemented by just keeping around old copies
of the buffer and reverting to them upon request, instead of the hair-raising
stuff emacs does.)
    millboss-imp.pkg can do similar things with its indices of running
panes and mills. All of this is very nice in a concurrent-programming
context because of the lack of required locking or such.  (In general,
concurrent programming is nicer in ML than in imperative languages
because ML has about 1% as many destructive heap updates, making for
about 1% as many bugs due to failure to adequately guard accesses to
mutable values.)

I'm sure there are still quite a few Mythryl design patterns which I have
not yet discovered, but I am at least getting to the point where I can
throw together many little projects pretty quickly just based on re-use
of existing code and familiar design patterns, instead of having to
invent that sort of stuff at just about every step.  That is a very
pleasant inflection point in the learning-Mythryl curve!



So anyhow, that's the status.  All that's left before doing a release
is finishing up the drawpane.pkg and millgraph-mode.pkg functionality,
both of which have all the scary parts done and only routine finishing
work left to do.   (Plus maybe get the documentation/website building
again --I haven't quite decided whether I want to spend any of my limited
remaining lifetime on that.)

I'd say that should happen by end of month, but between a crunch at
work and chemo fatigue eating my evenings and weekends a lot of late,
I'm no longer very confident of how much Mythryl programming time I
will be able to fit in a given calendar interval.



Life is Good!
 -Cynbe




Note[1]:  No, I'm not kidding.  In the commercial software development
world, a megabuck will buy you basically a few annual updates to gcc
involving routine patch application.  If you were to go out and get
a commercial quote for starting with SML/NJ and ending with Mythryl
where it is now, payment cash on the barrelhead on completion, any
quotes below $10M would most likely be fraudulent, in the hope of
negotiating additional money once the customer was on the hook. A
realistic, fair quote would probably be close to $100M.




More information about the Mythryl mailing list