Polishing D

Premise

The D programming language is a very good language, and many believe it to be the logical next step in native programming.  More and more people are using it, reviewing it, and talking about it; doing so even in a positive manner.

However, very often D is mentioned with a note that it has rough edges.  And this is true, it has many rough edges.  Even so, none of these rough edges are really crucial, so to speak - it is easy to get around them, ignore them, and never see them again.  It's hard to recognize such issues, and easy to cast them off as unimportant to real usage or real programmers.

The reality is, these rough edges scare away new programmers (whether with established experience with other languages or not.)  Many of these rough edges scare away specifically programmers used to some little thing in another language, not understanding that they can still have it with D.  Most importantly, these unimportant flaws tend to make people think there are bigger flaws.

Features of the language, and other issues already receiving attention, aren't the focus here.  Rather, things that seem unimportant or which aren't receiving attention.

These notes are coming from someone who has dealt with releasing software to a large population, with multiple branches under development, a couple dozen translations to the system, and support for the whole lot.  These are not empty suggestions, but rather suggestions based on experience.

The Compiler

The reference Digital Mars compiler is very good.  The GNU adaptation is also very good.  Not having another compiler built with a unique frontend makes the language feel less accepted, but in general is not a problem.

The compiler does, however, need to be better emphasized for how simply fast it is.  It is a common joke among programmers that compiling is one of the longest processes in development, but with DMD this is not true.  GDC may not share this speed, but it does show that the language allows for better simplicity.

This needs to be emphasized.  It alone can improve a programmer's development.

However, there are quite a number of rough edges here.  Specifically:

The Release Process

One thing people seem to note frequently is that GDC is not in sync with the DMD distribution, and often one step behind.  While, realistically, this is not a huge problem (most code works between many revisions of the compiler), it's also not even often true.  GDC seems to be, for the larger percentage of the time, up to date with the compiler.

In addition, releases of the DMD compiler have been made that did not function properly on a specific platform.  Even more, the DMD compiler is available only for Windows and Linux, which makes it harder to argue against people claiming it is not portable.

The main problems here all seem to center on the release process of the specification and of the reference compiler itself.  Some possible solutions to these problems include:

1. This has been suggested in the past, but it may be worthwhile to include DMD's frontend source in a read only source code repository, or to release frequent "nightly" builds of the compiler for reference or people watching development.
   

2. When possible, specification changes should be announced with a planned swag general release.  Even if the specification changes are not yet written or finalized, this gives other compiler vendors time to at least allocate resources to handle an expected release.
   

3. The compiler release system should be more heavily automated.  This is generally done in tandem with a source control system and a build system.  The latest version of the source can be pushed to a server, which then would build and package it in an automated fashion, which then leaves testing.
   

4. More people should "own" different aspects of the release.  This includes QA, management of the build system, review of the specification document changes, writing actual compiler changes, etc.  It appears (at least to the general public, whether this is true or not), that this is all being done by one person.  Both for scaling the project, and for public opinion, this needs to change.
   

5. The reference compiler should be released, if possible given time and backend restraints, on more platforms.  Automating the build system and adding people to specialize in various platforms would greatly improve the success of this.
   
   

Installation

Installing the compiler is easy.  Even putting it in your Program Files directory on Windows is fairly easy, but most people don't know how.  Installing it without privileges as a user on Linux is even possible.

However, this still has rough edges.  This is everyone's first experience with the language, and definitely the first time they can be scared off.  Many other languages have flaws here, and it shows.  Things like spaces in file names, terse configuration files, and etc. are a big drawback.

Solutions might include:

1. The compiler and linker should support, easily, installation and use with any paths the user might have.  This is especially important with Vista, where the compiler should not require Administrator privileges to install.  This is already true, but takes some work to fix.  Documentation is not the answer to these problems.  Making it easier is.  It should be accepted that on Windows, Program Files or even Documents and Settings is where people want it.
   

2. Although this seems unnecessary, it may make sense (especially with a build system), to provide an installer for D, whether command line or graphical.  It need not be complicated, but it can set path, ini, and other things for the user - to prevent any simple mistake from meaning someone does not use D.
   

3. Licensing either the compiler, or IDEs, to be bundled together may be of significant interest.
   

4. A debugger (WinDbg) is already included by default.  However, many know nothing of its existence or use.  While it's not the job of Digital Mars to document these things, and the windbg.html page already gives some documentation, it's really awareness that's a problem here.  A quickstart guide would work wonders.  Give an example of an access violation and debugging it.
   

5. The configuration files, dmd.conf and sc.ini, should be better documented.  What is "%@P%"?  This can be included in the config files.
   

6. Separate packages should be provided for Windows and Linux.  People are confused when they have files that are not useful to them.  Improving the build system would solve this.
   

7. Unlikely to be practical, but putting a test environment online would be of great use.  This might mean giving people a remote shell they can shell into and compile and test things on (with resets and privileges properly in place.)  Another possibility is a web interface, but this is slow and unlikely to be properly secure.
   
   

Testing and QA

Currently, the compiler is released in "alpha", "beta", and "release" stages.  Looking at a change log, or watching releases, it's difficult to tell what version is the correct one.  Many people feel more comfortable with the "good ol' stable", while many feel like it's less unless they're using the "latest and greatest."  Some people are interested in alpha testing and some people are not.

There is no testing process or anything exposed to the general public.  While the compiler still has a high quality standard, this doesn't give a good general impression.  Possible solutions:

1. More carefully separate downloads and releases into development phases, with more of an explanation (if terse) on why one might go for one or the other.
   

2. Consider using release candidates to preflight releases.  These might or might not be within a public circle, but either way should improve quality to the end user.
   
   

Digital Mars / Development Team

The perception of D is that it's still a more-or-less one man operation.  Irrespective of the greatness of this individual, it makes people worry about the language's future, ability to scale to growth, and well-rounded quality.

It is true that many people are well-involved with the project, but this should possibly be made clearer.  These people, potentially, might be given more official status or even positions within the company.  Whether the person is a volunteer or wants to be paid for their work really doesn't change this need.

Steps involved likely include:

1. Internally assessing what positions could be expanded and filled with other people.  This is a difficult observation to make from outside.  Also, there's no need for these to be full time positions.
   

2. Providing documentation on the website (in the form of credits) explaining who is involved, at least at the top levels.  The list need not be exhaustive, but rather provide understanding of specialization.
   

3. Giving a small set of people, interviewed and such, access to the full backend and training said people in various aspects, so that people with difficult skill or knowledge levels can skill troubleshoot, fix, and deal with problems in the compiler.  These people would still receive oversight by a lead, but should increase throughput of minor issues and polish significantly.
   

4. Moving to a source control system which might give different access to different people, and integrate into any possible build systems.  All checkins to this repository would likely be reviewed by a lead or at least a peer, to maintain quality.
   
   

Debugger and IDE

An old, Microsoft written debugger is provided with the DMD package.  An IDE of a sort is included with the DMC utility package.  These tools are possibly not ideal, and definitely not one a new user to D is going to find when downloading the compiler.

Most users of other programming languages, especially C/C++/C#, tend to consider the "full package" when comparing.  While this may not be "fair", at the end of the day it is practical and correct.

Nevertheless, including an IDE or debugger in D which is more modern might serve to stifle competition, or in the least cause some annoyance to developers of other systems.

The solutions in this case aren't as direct, but might be:

1. From a marketing and documentation standpoint, recognize this and be proactive in addressing it.  Perhaps note this right on the download page.  Users should not download the compiler without knowing they need to address debugging separately.
   

2. Improve possibilities for integration with IDEs, and address any deficiencies in parsing error messages from the compiler.
   

3. Consider providing the debugger package as a separate download.  Having it built in is nice, but if it's a separate link, people know that they are getting it.  This also reduces bandwidth for people who do not want it.
   

4. Encourage development of these tools.  Even if this is done by means of a competition, encouraged by bounty of users, making it official (and announcing it and pushing it officially) will increase its potential and raise people's incentives.
   

5. Provide better built-in debugging capabilities by adding stack tracing, when possible, to built in exceptions.
   
   

Documentation

There are many aspects to the documentation of D, and many rough edges here.  Arguably, this is very possibly the most important thing for a language to excel at, aside from its core quality.

D does not have bad documentation, but its documentation could use a lot of improvement.  Many other languages are better documented just by being more popular or having more money, but some do it better generally.

Phobos' Documentation

The quality of documentation for Phobos is about the same as for any other library.  This seems fine, but isn't when the core library of a language is being addressed.  A good comparison in this point is PHP.  Most will agree that its standard functions are very well documented.

Things that need to be improved include:

1. Many of the functionality in Phobos wraps functions already available in C.  It seems expected that the reader will know everything about the underlying C functions (even if they are not named.)  A good example of this is std.socket - terminology is undefined, explanations are terse, no examples are given, etc.
   

2. Module wide documentation (what it's for, terminology, basic use examples) is seldom if ever given.
   

3. Code examples should be provided for every method, or at least class and task.  Languages like PHP, C#, and even most custom frameworks already include this.  This takes time, but wouldn't be much of a problem with a person or people dedicated to the task (possibly separated by module.)
   

4. Comments from users aren't really available or at least visible.  Allowing users to discuss problems they are having with method X of class Y or function Z makes it easier to find answers and gives people confidence they will be able to get help.
   
   

The Specification

Having a language specification is important, and provides a window for people to implement good compilers.  The specification is a good read, and shows how simple and elegant D is.

However, a specification is not documentation.  Documentation is based more on tasks, quick starts, and on the end user.  A specification is meant for experienced people with ambitious objectives or long-term interest.

The solution here would seem to be simply providing more separation between documentation and specification.  While having them together is a good concept, it should not take away from people learning the language.

It's true that books can be written, but having something official on the website is crucial in these times.  There aren't many books and there are no classes in universities - people are going to come at D from a different angle.  It's also important to note that the documentation need not be as detailed or written by the same author, so long as it is correct, of good quality, and published officially.

Searching and Indexing

Other languages are not searched and indexed with Google.  Google is an amazing and popular search engine, but it's not the end all.  It is not an index.  It is meant for finding websites, not for finding specific paragraphs of information on a page.

The documentation needs to have its own, more direct interface for these things.  Organization is very important as well.  Without these, it becomes impossible for new programmers to find new features, making it as if they did not exist.

The need for this can be found evident by frequent posts to newsgroups about fairly simple tasks, which involve keywords or functionality not found in C.  It is important to remember that for every person who posts a question, four or five are not finding the answer.  Always look at frequent questions as defects, serious and important just as much as compiler segfaults on bad code.

Translation

Currently, translation is done much the same as with searching; by Google.  This is inevitably imprecise.  It is not difficult, when there is need, to find people interested in translating with reasonable quality.  While many can read English just fine, they may be able to understand complicated concepts (such as found in Phobos or the compiler) more quickly and easily when in their own language.

Underestimating the importance of internationalizing the compiler can be devastating.  A large portion of the programmers in this world do not speak English as their first language.

Support

Support is currently primarily given through newsgroups and email.  This is not ideal, and does not give people a perception of good support.  Providing good support for any large scale project is crucial to its success.

Some solutions include:

1. Lead people, after downloading, to an established community for support.
   

2. Whether the support is provided by a third party or Digital Mars itself, make sure it is accessible.  It should be recognized that newsgroups are not comfortable for many, and the current interface is both unfamiliar and difficult for a newcomer to easily find.
   

3. Recognize that while many programmers have egos, a lot are interested in paid support.  While this is likely already available, information about it is not as widely available as it is with other languages and systems.
   

4. Providing in-documentation support (from users) can be very effective and makes it easier for people who may not want to actually ask a question to find solutions more quickly and easily.  It also provides a way to encourage frequent questions to be folded into the documentation.
   

5. Consider tasking individuals (whether volunteer or paid) more officially with support.  This can have a larger impact than might be assumed.
   
   

Comparison with Other Languages

The comparisons with other languages could be improved and more accurately propose D's benefits and advantages.

It seems to "whine" that things are not part of the core language of C++, and thus are not part of it at all.  To many, this is unimpressive and a turn off.  Saying this is gaining no more users.

Some directions to consider:

1. It's important to note that being able to use strings as a core language feature makes them easier.  Instead of comparing with C++, compare with a language that does have them.  Make people remember the good times, not the bad.
   

2. Using D is better in part because it's not as time consuming.  Again, comparing it with other languages used for the same benefits is important - why do people use Python?  Why do people use other languages than C?  D is the answer to all of these problems without the drawbacks.  People know C has flaws, and that's why so many use other languages.  Better to explain why these problems are effectively solved.
   

3. The preprocessor is a big point.  This is known, and considered, but should not be forgotten.  Documentation needs to provide common tasks and their equivalents.  Answering this on the newsgroups is not enough.
   

4. Contract programming is important as well.  Many people do not know of these things.  Unit tests?  Explain their benefits and why D makes it easier than any other language to get into the habit of using them.... and why this is harder with other languages.
   

5. In general, consider the user base.  Is D taking users from C, or is it taking users who have already jumped ship but miss C for the flaws their particular language has?  This is absolutely crucial.
   
   

Phobos Itself

Phobos is not a bad library, but many are unsatisfied with it.  In general, it needs reorganization and consideration.  It feels incomplete to many, and that's because it simply is.

A group of people need to work together, and decide what should be included and what should not.  It needs to be decided what should be expanded upon, and what works for people.  What interfaces are cumbersome and which are not.  Whether another like Tango is simply to replace it or not.

And this needs to happen soon.  Right now, it feels like Phobos needs to change.  This makes people fear using it, because if it's going to change, their use of it will have to.  Make the change sooner than later so that people can feel comfortable with what's there, and know that they can use it.

Examples include:

1. Most of the sections within std.c are incomplete, especially std.c.linux.linux and std.c.windows.windows.  It is very frustrating to always look up manual pages and manually add these things, when they are supposed to be already there.  This is not insurmountable, it just requires someone to take it on and have responsibility of it.
   

2. Comparisons with other languages and with competitors need to be made.  Consideration of OOP and non-OOP methodology is a large area, but some people may not enjoy the style of Tango (or else, it would have likely long replaced Phobos.)
   

3. Looking into a centralized module registration system of some sort should be considered.  Many languages use this with great benefit, giving people an easy way to get to more advanced things like XML parsers that might not be included by standard.  C may not need this, because it is so popular, but D is not there yet.
   
   

Website

The website really needs to be expanded.  The simple truth is that it needs more people to have responsibility for it.  Many of the problems noted - support, documentation, searching, marketing in general, downloads, etc. are all website problems.

It may not be a bad idea to hire a person or company to look into some basic things for improving the website - in its design, layout, ease of use, and organization.

Interoptability

This is not an issue of importance, and is mentioned more for the purpose of showing the concept of providing more ways that D is unique than are currently done.

Interoptability with IDEs is usually handled this by piping the compiler and reading/parsing the input.

Providing an alternative interface to this could interest more developers of IDEs, would likely improve the quality of integration, or at the least improve the generation of new IDEs.  However this concept is likely not to be simple to implement.  It is simply proposed as a possible example of another way of making D unique in another field.

The DMD reference compiler could expose its functionalities through a DLL or shared object.  This library would have functionality to lexically parse code (most likely in a line by line fashion as editors typically work), semantically validate it, report errors in a manner that need not be parsed, and compile code directly without reading from a locally saved file.  These operations would have to be published as a separate standard, such that other compilers could provide similar interfaces, however.

Such an interface could also enable the compiler executable to give the user a selection of compiler version, e.g. dmd.1.0.12.dll, and would enable dynamic code compilation/execution within user code, providing for another native code solution to a common argument for interpreted/jit languages.

Other directions like this, even if they seem out of reach, impractical, or even silly, should be considered as a means to keeping D "modern."  Sometimes it can be a simple thing that makes a language popular, like with Ruby, even if the language itself should have stood on its own.