index.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="interface/layout.xsl"?>
<page>
<filename>/</filename>
<nextnode><link name="Introduction to Table-Oriented Programming" href="TOP/Introduction/What.xml"/></nextnode>
<content>
<p>If you're a developer, wouldn't you like to get more done?  And if you're a 
manager, wouldn't you like your developers to be more productive?</p>

<p>I would argue that in the 90s, individual developer productivity was higher 
than it is now.  Our goal here is to understand:</p>
<ol>
	<li>Why it was higher</li>
	<li>The (good) reasons it went downhill</li>
	<li>What we can learn to improve productivity</li>
</ol>

<h1>Why was it higher?</h1>

<p>The short version is that there was more productivity-oriented tooling; but 
exactly what was it?</p>

<h3>RAD IDEs</h3>

<p><i>Rapid Application Development - Integrated Development 
Environments</i></p>

<p>In the 90s, programming languages weren't just languages; they were entire 
toolsuites.  There were two most dominant general-purpose languages, C(++) and 
Pascal.  Each had its own tool suite.  </p>

<p>The example I'm most familiar with is Delphi.  This was made by Borland, and 
is a descendent of Turbo Pascal (if anyone is interested, there's an Open 
Source version called <a href="https://www.lazarus-ide.org/">Lazarus</a>).  </p>

<p>The big advantage here was that it allowed one to drag and drop components 
onto forms (or reports), edit the properties, attach them to databases, and 
then it created the backend code to make it all happen.  Want to know what 
happens when you click a button?  In the GUI editor, go to the Events tab, 
click the onClick event, and the code will appear in the code editor.  </p>

<p>These were not the only languages with an attached and integrated tool suite; 
MS Access is a classic example of one that's still like this.</p>

<h3>Domain-Specific Languages</h3>

<p>These were languages that specialised in particular areas.  There were a 
number that specialised in databases/tables; amongst these were SQL and the 
xBase family of languages, the second of these being one of the main 
inspirations behind TOP.  These languages had their drawbacks, but they were 
usually particularly good in the areas they specialised in.  </p>

<h1>Why Did it Go Downhill?</h1>

<p>From about 1995-2005, there was a lot of change happening.  </p>

<h3>Fragmentation</h3>

<h4>Fragmentation of Programming Languages</h4>

<p>From there being only a few languages, there was a great proliferation of 
languages; Javascript, Perl, PHP, Python, Ruby, and other scripting languages 
hit their stride around this point.  Competing more directly with C/Pascal, we 
had Java and C#.  </p>

<p>This meant that each language community put the majority of their effort into 
the language, and less into the tooling.  When there *were* efforts towards 
tooling, they were often multi-language, meaning that more languages got 
covered, but at the expense of other IDE features.  </p>

<p>Open Source got a big boost during this era too.  With so many langauges 
being available for free, people also, in my opinion, became less willing to pay 
for both languages, and IDEs that would make them more productive.  </p>

<p>Was this good?  Yes, we got some great programming languages out of it, and 
got people to rely less on C.  Was it worth it?  Yes.  </p>

<h4>Fragmentation of GUIs</h4>

<p>In the early 90s, each OS generally had one GUI, and that was what you 
developed for.  The primary GUIs were MacOS and Windows, with XWindows (Unix) 
also being in the mix.  </p>

<p>Then came the Web.  The web was not well-designed to be 
a GUI (especially in the early days), but that didn't stop people trying.  None 
of the GUI design tools worked for making web interfaces.  </p>

<p>Was this good?  Yes, because now when people wanted to use a program, they 
could just go to a website, instead of having to have a box with some disks and 
a manual mailed to them (or they'd have to go to the store to get it).  Was it 
worth it?  I'd say yes.  </p>

<h3>Scaling</h3>

<p>Programs changed from the days when most PC programs were made by developers 
sitting at home coding.  Suddenly there were issues of scale (which had been 
faced before, but often on corporate servers, rather than PCs).  </p>

<h4>Scaling of Concurrent Users</h4>

<p>With the advent of the web in 1995, suddenly many more programs were being 
run on centralised servers.  On the plus side, people didn't have to distribute 
their software any more; it was nicely there on the server.  On the minus side, 
this meant that programmers were suddenly having to worry about scaling their 
programs to multiple simultaneous users on the same computer.  </p>

<p>Was it a good idea?  Yes, because of all the things we can do with the 
Internet.  Was it worth it?  Yes.  </p>

<h4>Scaling of Developers</h4>

<p>Instead of a single person working on a program, there were now multiple 
people working on programs.  Source control programs existed (eg. RCS, SVN), but 
with the release of git in 2005, we finally had an Open Source solution for 
source control that was powerful and actually worked.  </p>

<p>Having multiple people working on a program introduces some overheads.  
Wanting to keep the production environment running while testing in a shared 
area introduces more overheads.  Think about the amount of time it takes you to 
make an one-line (or one-character) change on your local, vs. the amount of time 
it takes to commit it to development, staging and production in a useful 
way.  Then there's documentation and the like.  </p>

<p>Was this a good idea?  Yes, because now multiple developers can work 
together, and websites are less likely to go down.  Was it worth it?  Yes.  </p>

<h4>Scaling of Development Teams</h4>

<p>With the scaling of the team, UI designers were introduced.  This was 
definitely a good thing, but generally meant that there was one more person who 
had to be involved in the GUI design; it was no longer enough to drop some 
components on a form and call it done.  In some cases, the design tools weren't 
even fully accessible to the developers unless extra licenses were bought.  </p>

<h3>Inflexible Languages</h3>

<p>Many drawbacks to OOP have been pointed out over the years, and various OOP 
modifications have been made to overcome them (eg. roles, aspect-oriented 
programming, and the like).  A full-orbed OOP system is useful, but a poor 
one has its own scaling limits just like non-OOP programming does.  </p>

<p>Just as the "everything is a function languages" (like early PHP) found it 
more productive to add the object-oriented paradigm, so the "everything is an 
object (and only an object)" people were hampering their developers.  Unlike the 
other items in this list, this one is currently alleviated by the mere existence 
of multi-paradigm langauges (but how to get people to use them?)  </p>

<h3>Downhill Summary</h3>

<p>Were the things above worth it?  Yes.  Did the things above make our lives 
harder by reducing the available tooling?  Also yes.  What can we do about it?  
Read on!  </p>

<h1>How Can we Improve Productivity?</h1>

<div class="blockquote">

<p>Much that once was is lost, for none now live who remember it.  ... And some 
things that should not have been forgotten were lost. History became legend. 
Legend became myth. And for two and a half thousand years, the Ring passed out 
of all knowledge.  </p>

<div class="reference" style="display: flex">
<div style="display: inline-table">Galadriel (voiceover), <i>The Fellowship of 
the Ring</i> movie, Peter Jackson, Fran Walsh, and Philippa Boyens</div>
</div>
</div>


<p>There are still things we can learn from RAD IDEs and DSLs.</p>

<h3>What can we learn from Domain-Specific Languages?</h3>

<p>Before delving into DSLs, I'd like to make a very brief visit to:</p>

<h4>Multi-paradigm Programming</h4>

<p>Each programming paradigm is a tool in your toolbox.  The more paradigms a 
programming language allows for, the more tools you have in your toolbox.  
Each paradigm not available is one less tool in your toolbox.  </p>

<p>With this in mind, the best way to get the benefits of a DSL in the modern 
era is not to use the DSL, but to integrate the paradigm of the DSL into a 
language which is already multi-paradigm.  My personal preference is Raku (see 
<a href="TOP/Comparisons/Paradigm-Comparisons.xml">Paradigm Comparisons</a> for 
more information).</p>

<h4>Learning from DSLs</h4>

<p>No doubt there are many things we can learn from DSLs, but I'm not familiar 
enough with most of them to derive the lessons here.  I can highlight only one 
particular one we can learn from (that's not a markup language), and that is 
<b>Table-Oriented Programming</b>.  In the 1995-2005 era, Table-Oriented 
Programming (TOP) was argued to be an alternative all-encompassing paradigm to 
OOP.  Neither won this argument, but OOP was accepted in large part ("everything 
is an object, but maybe not <i>just</i> and object"), whereas TOP, to the extend 
it was used at all, was left to SQL and spreadsheets.  But the lessons that TOP 
has can be applied in a multi-paradigm setting without disrupting the other 
paradigms, and will improve our data handling.</p>

<h3>What can we learn from RAD IDEs?</h3>

<h4>Lessons from RAD IDEs</h4>

<p>The lesson here is easier for me to identify, probably because I've spent 
more time over the years using various RAD IDEs.  The big lesson is, in many 
cases, visual editors of various sorts make people more productive.  I think I 
can best express this by making a bullet list of the things that I think should 
be implemented in an IDE that fully supports Programming Productivity:</p>

<ul>
	<li><b><a href="RAD/Database-Design-Editor.xml">Database Design 
		Editor</a></b>: Lets you visually edit the design of your database</li>
	<li><b><a href="RAD/Program-Editors.xml">Program Editors</a></b></li>
	<ul>
		<li><b>Regular Program Editor:</b> These are standard; no explanation 
			needed</li>
		<li><b>Source control visual tools:</b> To reduce the time spent managing 
			the code</li>
		<li><b>Relation Code Editors:</b> In Table-Oriented Programming, both 
			code and data can be stored in tables.  Think of this as being like 
			a spreadsheet, but you can compile it.</li> 
		<li><b>Dataflow Code Editors:</b> In dataflow programming, there's a 
			diagram that represents how the data flows.  Some of the nodes in 
			the diagram have code inside them.  We should be able to edit it.  
			</li> 
	</ul>
	<li><b><a href="RAD/UI-Editors.xml#GUI-Editor">GUI Editor</a>:</b> Something that allows editing the GUI in a visual, 
		drag-and-drop manner (Forms)</li>
	<li><b><a href="RAD/UI-Editors.xml#Document-Editor">Document Editor</a>:</b> This allows people to edit a document that could 
		be made into a report.  This would be influenced by word processors, 
		and by the Notebook model (cf. Jupyter)</li>
</ul>

<h3 id="UI-churn">UI Churn/Volatility</h3>

<p>One final lesson -- this isn't from RAD or DSLs, but from the SmartDB 
advocates.  </p>

<div class="blockquote">

<img src="TechnologyStackStabilityJava.png" style="width: 80%; margin: auto; 
display: block"/>

<div class="reference" style="display: flex">
<div style="display: inline-table">Slides for <i>The Database: Persistence Layer 
(NoPlsql) or Processing Engine (SmartDB)?</i>, Toon Koppelaars</div>
</div>
</div>

<p>In the talk referenced, Toon Koppelaars made points similar to the following:
</p>
<ul>
	<li>The closer the technology is to the user, the higher the technology 
		churn rate.  The closer it is to the database, the lower the technology 
		churn rate.  The image given specifically references Java, but his 
		slides showed that the rate for Javascript was even higher.  On the 
		other hand, the closer the technology is to the database, the lower the 
		technology churn rate.  </li>
	<li>If layers in your chosen technology stacks are volatile, then you ought 
		to use them "thinly".  Do not do business logic in them; instead, push 
		business logic further down into code-stack where stable layers exist.  
		Why? Because the top-layers change more quickly than does your business 
		logic.  If you don't, you'll end up with a maintenance nightmare (ie. 
		old technology that doesn't upgrade well)</li> 
	<li>UI and Database developers have quite different focuses:</li>
	<ul>
		<li>Database developers focus on: correctness, maintainability, 
			performance and scalability (solid)</li>
		<li>UI developers focus on: extremely friendly UI experience (UX) 
			(hip, trendy)</li>
	</ul>
	<li>Organisationally, Backend and Frontend must co-operate.  Database 
		developers will need to understand UI-developers' world, but NOT <i>vice 
		versa</i></li> 
	<ul>
		<li>Frontend will be "decoupled" via API's</li>
		<li>Frontend only need to know enough architectural concepts to 
			communicate effectively and jointly design API-layer</li>
	</ul>
	<li>Thus, database developers, <b>must</b> be in the lead.  Why?  They own 
		the database design and business logic</li>
</ul>

<p>This is why I added a database design tool to the list of editors</p>
</content>
</page>