Here are my notes from the software portal talk Pascal and I gave yesterday at the openSUSE conference.
The slides we used are also available.
What is the Software Portal?
Software Portal is a free-software web-application that runs as a service on on opensuse-community.org. It provides both a web-ui and an API. It is possible you are already using the software portal if you use the webpin search page or the command line webpin programme written by Pascal. For the last few months this has been backing on to the software portal.
The software portal indexes package repositories, both those discoverable with the openSUSE build service API and those users add manually through the web interface. It also allows users to contribute additional metadata such as ratings, comments, tags, and screenshots. These two sources of information are both associated against “applications”. The web user-interface is primarily a way of finding, installing, and adding information about applications.
Applications
So, what is an application? We define applications within software portal to be a higher level concept than packages. Applications are what users perceive software as – A product they are looking for and may wish to install to perform a task. Packages are the mechanism of how applications are distributed to users. Packages are platform-specific, they require different packages for different operating systems, and different architectures. The split between packages is also defined by pragmatic issues related to distribution more than conceptual boundaries. Some applications may require multiple packages installed in order to use. Conversely some packages contain more than one application. For example the application GIMP on openSUSE on a 32bit platform might consist of the main gimp package, a gimp-banding package, and some core plugins in another package. Another opposite example is the kdenetwork3 package that contains multiple chat applications.
The web-ui provides application-level search, tag-browsing, and installation.
In the software portal we strive to map packages to the appropriate applications. There are three mechanisms we currently use to do this:
- A set of rules stored in the database that map common package naming conventions to application names.
- The presence of .desktop files in packages.
- User-defined mappings.
For example: currently kde4 applications tend to have the packagename kde4-packagename. Many non-desktop specific packages start with the application name. Rules are assigned priorities so that more-specific rules can override general rules.
Through the web-ui, users can define additional applications and edit existing applications. While editing, users can define rules for which source-packages should be mapped to which applications during repository indexing.
One of the main aims of abstracting away from packages to applications is that we want to allow users to just click install at an application level, without having to know what their hardware/software platform is or choose which package vendor to use. To this end we need some way to choose a preferred vendor for each application in the case where there are multiple vendors of that application. We do this by
- Associating a priority with repositories based upon their support status and stable/experimental status.
- Allowing power-users to customise the preferred vendor and selection of binary packages to install by default for an application. Via the web-ui.
So the highest priority would be the update and static distribution repositories, and the lowest would be random home: repositories in the openSUSE build service.
The operating system agnosticism is provided by one-click-install and the system package management can choose the appropriate architecture packages.
Why is it needed?
openSUSE embraces third party vendors of packages. Which means that there’s a low barrier to entry and anyone can start packaging and publishing software to users. However, this freedom leads to a confusing situation for users.
Many repositories
(Software Portal is currently indexing around 4000 and this is not exhaustive.)
Package duplication.
Some applications such as amarok has been packaged dozens of times for openSUSE by different people for various reasons. In this case how does a user choose which vendor to choose? We also encourage duplication by not making it easy to find existing packages.
Lack of user-feedback
One packager may package dozens of packages but may not extensively use many of them. We’re not currently good at collecting feedback of package and software quality from users. It’s also not clear for users where to file bugs for the software they’ve just installed.
Confusing concepts
It should not be necessary for users to understand concepts such as packages and package repositories just to install software. It’s also unnecessary for users to know what operating system (version) they’re running or on what hardware.
Insufficient data
We currently rely heavily upon a few package maintainers to source and supply metadata for packages. Packages come with descriptions and summaries but lack more rich metadata such as screenshots. There are many more end users than packagers so it makes sense to let them contribute this information.
Implementation Platform
Software portal project is implemented using Java 6, making heavy use of Spring Framework, Apache Wicket, Jersey, Maven, and MySQL.
Future Plans
We are currently working on finalising a “1.0” release, fixing some of the major bugs, performance issues, and polishing the user interface to get to the point that it is safe for end users to start using and we can publicise it. We need help right now in testing and finding bugs.
The biggest performance issues I am currently working on are:
- Search of all 20+ million unique file names every time someone performs a search.
- Calculation of the best binary packages to install for applications with dozens of different vendors.
Challenges arise from the frequency of change of the package repositories (thousands of packages may change every day). Index formats are a balance between query performance and update performance.
After first release one of the biggest issues we want to resolve is that people cannot use their existing login that they use for all other openSUSE services to log into the software portal. Due to Novell control of the authentication system. We have a dependency on Novell here, but hope to implement openID support and that Novell will be able to implement an openID provider.
Further ahead we would like to work on things like:
A recommendation engine
We could do things like recommend you plugins for software you have installed.
User-specific search.
When we know information about the user, their preferences, and what operating system they have installed we can tailor search results to match them, prioritise applications that are available for their platform, and give higher priority to applications for their preferred desktop.
Translation
The web-ui is already translatable. Application descriptions are stored in a translatable manner but the source of this information (package repositories) generally does not have translations. We could potentially build an interface to allow people to translate the content itself if there were a demand for it.
Use Build-Service remoting or Hermes notification framework
Move towards a push model where the software portal is notified of new software, rather than the current pull system where we have to poll package repositories to see if they’re changed, then compute the differences from the previous version if they have.
Integration with bugzilla
Report bugs directly from software portal.
Integration with upstream
Perhaps via the DOAP (Description of a Project) format. We could both consume DOAP files provided by upstream projects, and expose DOAP for software we have indexed.
The Master Plan
Our original aim was to supplement the web-service with a rich desktop client that utilises the information indexed by the software portal and combines that with information available to it as an application running on the local system, such as the currently installed packages, and the repositories the user subscribes to.
Perhaps a YaST module? We already have a package-search yast module that does use the software-portal API to search for packages, but much more powerful things are possible. Unfortunately we do not currently have the time to work on a desktop client on top of the web app itself so if anyone is interested in working on this side of things please get in touch.
Demo
We gave a demonstration of some of the features mentioned above during our talk. You can play with the current version online. Though please be aware there are lots of known bugs and performance issues, it’s not ready for end-users yet.
That said, please do play with it, and let us know what’s broken by filing bugs under the openSUSE.org->Software Portal bug component.
If you want to try out more of the administrator features such as adding repositories, adding applications, customising package to application mapping etc then create an account and drop me an email so I can grant you permissions.
How can you help?
- Start testing it
- File bug reports
- Help create and organise content
- Developers who know Java and want to contribute code would be welcome. We would be happy to help people get started with the codebase.
- Designers can help us improve the look and feel and interaction. Some of the processes such as mapping packages to applications are somewhat klunky. It would be good to get some suggestions on how to make this functionality easier to use.