Delphi/Rad Studio desperately needs a proper package/library/component manager. A package manager provides a standardized way of consuming third party libraries. At the moment, use of third party libraries is very much adhoc, and in many cases this makes it difficult to move projects between machines, or to get a new hire up and running quickly.
Other developement environments, like the .net and javascript eco systems, recognised and solved this problem many years ago. Getting a .net or javascript project up an running, in a new working folder or new machine is trivial.
With Delphi/Rad Studio, it's much harder than it should be. In consulting work, I've made it a point to see how clients were handling third party code, and every client had a different way. The most common technique was... well, best described as adhoc (with perhaps a readme with the list of third party products to install). Getting that code compiling on a CI server was a nightmare.
Existing Package Managers
Embarcadero introduced their GetIt package manager with XE8, and the GetIt infrastructure has certainly has made the installation of RAD Studio itself a lot nicer. But as a package manager for third party libraries, it comes up short in a number of areas.
There is also Delphinus, which is an admirable effort, but hasn't gotten much traction, possibly due to it being strongly tied to github (you really need github account to use it, otherwise you get api rate limiting errors).
Rather than pick apart GetIt or Delphinus, I'd like to outline my ideas for a Delphi package manager. I spend a lot of time working with .net (nuget) and javascript (npm, yarn), so they have very much influenced what I will layout below.
I have resurrected an old project (from 2013) that I shelved when GetIt was announced, and I have spent a good deal of time thinking about package management (not just in Delphi), but I'm sure I haven't thought of everything, I'd love to hear feedback from people interested in contributing to this project, or just potential users.
Project Ideals
These are just some notes that I wrote up when I first started working on this back in 2013, I've tried to whip them into some semblance of order for presentation here, but they are just just rough outline of my ideas.
Open Source
The Project should be Open Source. Of course we should welcome contributions from commercial entities, but the direction of the project will be controlled by the community (ie users). The project will be hosted on GitHub, and contributions will be made through Pull Requests, with contributions being reviewed by the Steering committee (TBA).
Public Package Registry
There will be a public website/package server, where users can browse the available packages, and package authors can upload packages. This will be a second phase of the project, with the initial phase being focused on getting a working client/package architecture, with a local or network share folder for the package source.
The package registry should not be turned into a store. Once a public package registry/server is available, evaluation packages could be be allowed, perhaps by providing a fee (web hosting is not free). Commercial vendors will of course be able to distribute commercial packages directly to their customers, as the package manager will support hosting of packages in a shared network or local directory. Package meta data will include flags to indicate if the packages are commercial, eval or free/open source. Users will be able to decide which package types show up in their searches.
Package Submission
Package submission to the public registry should be a simple process, without filling in and signing and faxing of forms! We will follow the lead of nuget, npm, ruby etc on this. There should be a dispute process for package names, copyright infringement etc. There will also be the ability to assign ownership of a package, for example when project ownership changes.
Package Authors will be able to reserve a package prefix, in order to prevent other authors from infringing on their names or copyrights. For example, Embarcadero might reserve Emb. as their prefix, TMS might reserve TMS. as theirs. (of course I'm hoping to get both on board). The project will provide a dispute resolution process for package prefixes and names.
Delphi specific challenges
Delphi presents a number of challenges when compared to the .net or nodejs/javascript world.
Compatibility
With npm, packages contain source (typically minimized and obfuscated) which is pure javascript. Compatibility is very high.
With Nuget, packages contain compiled (to .NET IL) assemblies. A package might contain a few different versions, that target different the versions of the .net framework. Again, compatibility is pretty good, an assembly compiled against .net 2.0 will work on .net 4.7 (.net core breaks this, but it has a new compatibility model, netstandard).
If we look at Delphi, binary compatibility between Delphi compiler versions is pretty much non existent(yes, I know about 2006/7 etc). The dcu, dcp and bpl files are typically only compatible with the version they were compiled with. They are also only compatible with the platform they were generated for (so you can't share dcu's between 32 and 64 bit windows, or between iOS and Android). So we would need to include binaries for each version of Delphi we want our library to support. This also has major implications for library dependencies. Where as npm and nuget define dependencies as a range of versions, a binary dependency in Delphi would be fixed to that specific version. There is a way to maintain binary compatibility between releases, provided the interfaces do not change, however exactly what the rules are for this is hard to come by, so for now we'll ignore that possibility.
That limits the scope for updating to newer versions of libraries, but that can also be overcome by including the source code in package, and providing on the fly compilation of the library during install. My preference would be for pre-compiled libraries, as that speeds up the build process (of course, since that's an area I have a particular interest in). In Continuous Integration environments, you want to build fast and build often, rebuilding library code with each CI build would be painful (speaking from experience here, 50% of time building FinalBuilder is building the third party libraries).
There's also the consideration of Debug vs Release - so if we are including binaries, compiled for Release would be required, but Debug optional? The size of a package file could be problematic. If the package contains pre-compiled binaries for multiple compiler versions, it could get rather large. So perhaps allow for packages that either support a single compiler version, or multiples? The compilers supported would be exposed in the package metadata, and perhaps also in the package file name. Feedback, ideas around this would be welcome.
Package files would be (like with other package managers), a simple zip file, which include a metadata (xml) file which describes the contents of the package, and folders containing binaries, source, resources etc. Packages will not contain any scripts (ie to build during install) for security reasons (I don't want to be running random scripts). We will need to provide a way to compile during install (using a simple dsl to describe what needs to be done), this still needs a lot of thought (and very much involves dependencies).
Library/Search Paths
Say goodbye to the IDE's Library path. It was great back in 1995, when we had a few third party libraries and a few projects and we just upgraded the projects to deal with library versioning (just get on the latest). It's simply incompatible with the notion of using multiple versions of the same libraries these days.
I rarely change major versions of a library during the lifespan of a major release of my products, I might however take minor updates for bugfixes or performance improvements. The way to deal with this is simply to use the Project Search path. Project A can use version 1 of a library, Project 2 can use version 9, all quite safely (design time components do complicate this).
Where a project targets multiple platforms, installing a package should install for all platforms it supports, but it should be possible for the user to specify which platforms they need the package installed for.
Design time Component Installation
The Rad Studio IDE only allows one version of a design time package to be installed at a time. So when switching projects, which might use different versions of a component library, we would need a system that is aware of component versions, and can uninstall/install components on the fly, as projects are loaded.
I suspect this will be one of the biggest project hurdles to overcome, it will requires someone with very good open tools api knowledge (ie, not me!).
Dependencies
Libraries that depend on other libraries will need to specify those dependencies in a metadata file, such that they can resolved during installation. As I mentioned above, binary compatibility issues make dependency resolution somewhat more complicated, but not insurmountable. The resolution algorithm will need to take into account compiler version and platform. The algorithm will also need to handle when a package is compiled from source, for example, binary only packages should not be allowed to depend on source only packages (to ensure compatibility). If we end up with install time package compilation, then some serious work will be needed on the dependency tree algorithm to work our what else needs to be done during install (ie, do any dependencies need to be recompiled?).
This is certainly more complicated than other platforms, and a significant amount of work to get right (ps, if you think it isn't, you haven't considered all the angles!)
General Considerations
Package Install/Restore
The user should be able to choose from a list packages to install. When installing the package, this would be recorded either in the dproj, or a separate file alongside the drproj. The install process will update the project search paths accordingly. Package meta data would control what gets added to the search paths, my preference would be for 1 folder per package, as that would keep the search path shorter which improves compile times.
When a project is loaded, the dproj (or packages config file) would be checked, and any missing packages restored automatically. This should also handle the situation where a project is loaded in a different IDE version.
Security
We should allow for signing of packages, such that the signatures can be verified by the client(s). Clients should be able to chose whether to only allow signed packages, or allow signed and unsigned, and what to do when signature verification fails. This will allow users certainty in the authenticity and integrity of the package (ie where it comes from and whether it's been modified/tampered with).
Clients
It is envisaged that will be at least 2 clients, a command line tool, and a Rad Studio IDE plugin. Clients will download packages, add those packages to project/config search paths. A local package cache will help with performance, avoiding repetitive package downloads and also reduce disk space demands. The clients will also detect available updates to packages, and package dependency conflicts.
Command line Client
The command like tool will be similar to nuget or npm, which provide the ability to create packages, install or restore missing packages, update packages etc. The tool should allow the specification of compiler versions and platforms, as this is not possible to detect from the dproj alone. This is where the project is currently focused (along with the core package handling functionality).
RAD Studio IDE Client
An IDE plugin client will provide the ability to search for, install, restore, update or remove packages, in a similar manner to the Nuget Visual Studio IDE support (hopefully faster!). This plugin will share the core code with the the command line client (ie, it will not call out to the command line tool). I have not done any work on this yet (help wanted).
Delphi/Rad Studio Version Support
Undecided at the moment. I'm developing with XE7, but it's possible the code will compile with earlier versions, or be made to compile with minor changes.
Summary
Simply put, I want/need a package manager for Delphi, one that works as well as nuget, npm, yarn etc. I'm still fleshing out how this might all work, and I'd love some feedback, suggestions, ideas etc. I'd like to get some people with the right skills 'signed up' to help, particularly people with open tools api expertise.
Get Involved!
I have set up a home for the project on GitHub - The Delphi Package Manager Project - RFC. We'll use issues for discussion, and the wiki to document the specifications as we develop them. I have created a few issues with things that need some dicusssion. I hope to publish the work I have already done on this in the next few days (needs tidying up).