Mica Development Guidelines

Version History

Revision

Date

By

Comments

0.1

2009/02/18

Ed Swartz

Initial draft, several sections missing still (coding style, etc.)

0.2

2009/02/19

Ed Swartz

More notes about new plugins/features, feature dependency order, coding style, and wired up TOC

0.3

2009/05/18

Ed Swartz

Add API control notes to the “development” section, documentation notes to the “finishing” section

0.4

2009/05/21

Ed Swartz

More notes about adding new features and plugins



Table of Contents

Version Control Policy

Development Setup

Maintaining Branches

Development Practices

Adding New Plugins or Features

Finishing a Feature or Bugfix

Bug and feature databases



Version Control policy

Version control system

In Mica, we use Subversion repository as hosted by garage.maemo.org.

The base URLs for the repositories for each project are:

https://garage.maemo.org/svn/mica/...

https://garage.maemo.org/svn/esbox/...

https://garage.maemo.org/svn/pluthon/...

https://garage.maemo.org/svn/pc-connectivity/...

(Don't check these out directly – use trunk or a branch!)

Your garage.maemo.org account name and password are used to authenticate commits.

Branching model

The sources use a two-tiered branching model which allows development to proceed without affecting the stability of the product.

The branching model that follows is based on this document: http://www.infoq.com/news/2008/04/kniberg-agile-version-control

The project source bases in SVN (mica, esbox, pluthon, and soon pc-connectivity, we hope) have a trunk and several branches.

The trunk holds stable, tested, releasable code. For any official or automated build, the sources come from trunk. Commits never occur directly to the trunk except in rare cases (e.g. fixing the automated build, bumping build numbers, etc).

Branches are created freely and have a less strict policy. Usually each developer makes a branch. If more than one developer is working on the same task, though, they should probably share a branch to make the workflow easier.

Changes that go into the trunk should be merged into your branch as soon as possible, to ensure you're always developing against current sources.

When a feature or bugfix is finished, the changes should be merged into the trunk and integration tests should be run locally before marking the feature or bugfix “fixed”.

Details about the mechanics of this process follow later.

Development Setup

Build environment

The ESbox and PluThon products and the Mica framework uses a common build environment.

The build environment has two pieces: the base environment, as an archive containing a full installation of Eclipse and required features, and external sources, as additional projects checked out from Subversion.

See http://mica.garage.maemo.org/setup.html for the instructions about setting up the environment.

Check out the external projects from https://garage.maemo.org/svn/mica/build/external/ and import these into your workspace as well. These are dependencies of Mica. Prebuilt plugins should not be used for any of these, since we have local changes.

Occasionally, plugins from the base environment will be migrated into the build/external directory as we find bugs that are not fixed upstream (or which are fixed only in new versions that we cannot use yet). This will require you to manually remove the respective plugins from the base environment – either uninstalling from within Eclipse or deleting them on disk – and fetching new external sources.

Always consult the README.txt in the buildenv/fullbuild directory for notable changes. The environment is huge so we do not update it often, so there are sometimes manual changes you must make.

Eclipse setup

We use Java 1.5 / 5.0 SE for the compiler. You may run the environment with a newer JDK, but ensure that the workspace has an “Execution Environment” set up for building. This will ensure that code does not get populated with Java 6 annotations which will break the build.

See Window > Preferences > Java > Installed JREs > Execution Environments and be sure the “J2SE-1.5” entry refers to a 1.5.0 JDK.

The Window > Preferences > Java > Compiler > Compiler compliance level setting should be 1.5.

All new files must have the proper license in the header. Ensure you have a File template like this in Window > Preferences > Java > Code Style > Code Templates > Comments > Files:

/*******************************************************************************

* Copyright (c) ${year} <your organization>

* All rights reserved. This program and the accompanying materials

* are made available under the terms of the Eclipse Public License v1.0

* which accompanies this distribution, and is available at

* http://www.eclipse.org/legal/epl-v10.html

*

* Contributors:

* <you> (<your organization>) - initial API and implementation

*******************************************************************************/


Also, it's generally good, if you make big changes to sources, to add yourself to the Contributors line in a file as well (with a note like “refactoring” or “bugfix” or whatever).

The Window > Preferences > Java > Code Style > Formatter settings should be the default (use the “Eclipse [built-in]” profile).

The Window > Preferences > Java > Code Style > Organize Imports settings should be the default (ordering of “java”, “javax”, “org”, and “com”, with 99 imports before using “*”).

Creating your branch(es)

To start development on a project, for which you intend to make commits, make a work branch for each project like this:

$ svn copy -m "Create work branch" https://garage.maemo.org/svn/<project>/trunk https://garage.maemo.org/svn/<project>/branches/work_<myname>

You don't need a branch until you actually develop on a project. Before then you can build against the trunk.

NOTE: Yes, branch the entire project. Branches are cheap in Subversion, and you'll almost certainly need to touch more than one plugin at a time.

NOTE: Please do not create branches inside Eclipse – the two major SVN plugins do not understand how to make single branches for the entire project. Instead they will make a lot of branches, one for each project, and this will be very painful to use.

Workspace setup

Keep a copy of the external sources checked out. I recommend a keeping development and trunk sources separate, e.g. “mica-devel” (for your branch, if used), “mica-trunk” (for the trunk), etc. for ESbox and PluThon as well.

$ mkdir sources
$ cd sources
$ svn checkout https://garage.maemo.org/svn/mica/build/external mica-build-external
$ svn checkout https://garage.maemo.org/svn/<project>/branches/work_<myname> <project>-devel  # if developing
$ svn checkout https://garage.maemo.org/svn/<project>/trunk <project>-trunk
... for mica, esbox, pluthon ...

Make two Eclipse workspaces, one for development and one for integration testing.

Use File > Import > Existing Projects Into Workspace to get the projects into your workspace.

Import the same external sources into each. Import “devel” projects into the development workspace (or “trunk” projects if you don't use a branch) and “trunk” projects into the integration testing workspace.

It's best to have all the related product sources checked out (ESbox and PluThon and Mica) -- don't use previous releases of the product as a Target Platform. This project evolves rapidly and we haven't yet specified any APIs, so a target platform will not be useful yet.

When running or debugging the product, create the launch configuration from the ESbox.product or PluThon.product files (open in an editor and use the Run/Debug icons in the editor toolbar). This will ensure you're running with as similar an environment as possible to the end-user product.

Subversion and Eclipse

You may use Subversion in Eclipse, via the Subclipse or Subversive plugins. My preference is Subclipse – I've had occasionally very irritating problems with Subversive that make it necessary to resort to manual patches to fix up problems, especially after refactoring.

As above, the projects should be checked out once and then imported into the workspace. It's possible to check out from Eclipse, but these plugins aren't very smart and can't always tell that all the projects are in the same repository. Then your commits will likely arrive in a whole lot of tiny pieces, and most all SVN operations will be slower.

TIP: You can commit from Eclipse, and this will usually result in single batched commits, but in both plugins, it's much faster to perform update and merge operations from the command line. You need to Refresh your workspace after doing this. I usually have “Window > Preferences > General > Startup and Shutdown > Refresh workspace on startupturned on to help with this.

TIP: If you need refactor a class by moving it across projects (esbox to mica, etc), neither plugin does this very well with the “Move class” refactoring, presenting a confusing error message instead. You need manually cut and paste the class in this case.

Unit testing setup

The two major test suites are ESboxProductTestSuite and PluthonProductTestSuite. These should be launched with ample memory (add “-Xmx512M” to the virtual machine arguments if using a Sun JVM).

TIP: to speed up tests, define a SCRATCHBOX_CACHE environment variable in your unit test's launch configuration and set it as to some file (e.g. “/tmp/scratchbox_cache.dat”). This will prevent the tests from re-scanning your installation every time you run them.

These tests rely on the ability to find a build machine to contact for performing SSH tests.

Please open the class org.maemo.mica.common.core.tests.TestMachineUtils and read its Javadoc. It will help you configure the reference to this build machine, which is in the form of a URI, and stored as a file on your system or passed as a system property on launch.

On Windows or OS X hosts, these tests need to create and launch a virtual machine. TestMachineUtils will also tell you how to specify a URI to launch a QEMU or VMware machine. You can run this class as a Java application to help you encode the password which must appear in the URI.

Also, on these hosts you generally want to handle mounting shared folders without user interaction These make use of some external text files which contains your authentication information. Run org.maemo.mica.internal.common.core.tests.UnitTestSharedFolderCredentialsProvider as a Java application and it help you add entries to the file and encode your password, so it will not be easily stolen.

Maintaining Branches

In the Maemo Eclipse Integration project, we use agile and scrum as a development methodology. And, as mentioned above, the branching model follows the document in http://www.infoq.com/news/2008/04/kniberg-agile-version-control.

Commits in your branch(es)

Generally, your branch is your own sandbox and you define the policy on the branch.

For the benefit of others on the team, especially those who might need to review your code or who might want to try it out, I recommend using meaningful commit comments to indicate your opinion about the state of the branch. For example: “Initial commit for ...” or “Checkpoint commit ...” or “Implemented feature / Fixed bug ...”. Empty commit messages are almost always a bad idea.

Also, it's a good idea, even in a longer-term feature, to keep the code on your branch ready to release, especially if you're near the end of a sprint.

Daily merge process

Merge changes from more stable branches into your release as soon as possible. (Currently, the trunk is the stable release and your branch is your release. You're working on your branch almost all the time.) Follow commit messages on mica-commits/esbox-commits/pluthon-commits mailing lists to see when this is needed.

It's not strictly required to merge every day, but it's ideal if your changes are in step with the trunk, so you don't have a lot of merging to do before your own merge with the trunk (see below).

EXAMPLE (adapted from http://svnbook.red-bean.com/en/1.1/ch04s04.html).

Assume that you last committed to your branch at rev 498. You're up-to-date with trunk changes already merged into your branch.

Overnight you see a commit rev 500 on the trunk. Do:

  your-branch$ svn update
  At revision 500.
  your-branch$ svn merge -r498:500 https://garage.maemo.org/svn/project/trunk
  A ...
  M ...
  D ...
  your-branch$ <<CHECK CONFLICTS: see below>>
  your-branch$ svn commit -m "Merged rev 498:500 from trunk."
  Committed revision 501.

When conflicts arise, they should only be due to changes you made for your story. If there are superfluous changes (formatting, line endings, etc)., then the trunk's version wins. If you're running from the command line, as above, use the “tf” (theirs-full) option when prompted how to resolve conflicts.

Remember: other people may have branches open against the trunk and will be merging against it as well. Make it easier for them by having source that's as similar as possible to the trunk, otherwise they will have to resolve all the same conflicts (in the other direction) when they "copy to trunk".

Continuing... you commit a few changes that day, at revs 502 and 503. The next day, you see a few more commits to the trunk, at revs 504 and 505. Run svn merge from the last point where you merged, again:

  your-branch$ svn update
  At revision 505.
  your-branch$ svn merge -r501:505 https://garage.maemo.org/svn/project/trunk
  A ...
  M ...
  D ...
  your-branch$ <<RESOLVE CONFLICTS>>
  your-branch$ svn commit -m "Merged revs 501:505 from trunk."
  Committed revision 506.

(The range of revisions overlaps your commits, but they were on another branch, so they don't interfere with anything.)

TIP: if you are merging a single commit, you can use “svn merge -c<revision> <URL>”. The exact revision is specified, so it's easier to handle.

If you're unsure where your last commit was, use “svn logto search for your commit messages.

Recreating your branch

Subversion is pretty cool, but some accidents or out-of-order merges on your side may lead to a situation where your branch always encounters conflicts when you merge, or even worse, causes SVN to issue scary error messages with lots of UUIDs embedded in them. (We've all seen these a few times, unfortunately!)

If this happens, wipe out and recreate your branch. Ensure you have any pending commits already on the trunk or saved off into patches, then recreate the branch like this:

$ svn remove -m "Remove work branch" https://garage.maemo.org/svn/branches/work_<myname>
$ svn copy -m "Create work branch" https://garage.maemo.org/svn/mica/trunk https://garage.maemo.org/svn/branches/work_<myname>

Development Practices

Code reuse

User friendliness

User interface guidelines

Testing guidelines

Coding style

Generally, follow the Eclipse developer guidelines (http://wiki.eclipse.org/index.php/Development_Conventions_and_Guidelines).

In Mica and its products, these general guidelines should be followed:

API Correctness

If developing inside Mica, be careful about you expose to the 3rd-party developer as API. Mica is a framework and should be relatively stable and not have unexpected API breaks.

Package Naming

org.maemo.mica.internal.api: packages in this hierarchy may be used only by other Mica plugins or by ESbox and PluThon (or products to which we have write access). Classes and interfaces here are expected to change, be renamed, appear and disappear over time.

org.maemo.mica.internal: packages in this hierarchy may never be used outside the owning plugin, except by unit tests.

org.maemo.mica: anything else is API.

API references in API

Note that the transitive closure of identifiers referenced in API (any type passed as an argument, returned or thrown by a method, any type of a public member or constant) must be API as well:

package org.maemo.mica.awesome;
public interface IMyInterface {
     MyConstant constant = ...;
     AwesomeReturn doSomething(AnotherType t);
     void fail() throws AwesomeException;
};

Contents of initializers, or types inside method bodies, do not have to be API. (The intent, of course, is that a client can actually use the API; it's less important that he can reimplement a method using the original code.)

This means that the scope of API can grow quickly. Favor interfaces, in general, to avoid exposing concrete implementation classes. Sometimes you must reference a type and promote it to API, but do not want clients to extend or implement it, just reference it. Use @noextend or @noimplement tags in this case.

Interfaces vs. Classes

For forward compatibility, you should not generally provide interfaces that the client must implement, with the exception of common cases like listeners, callbacks, etc.

The reason for this is, if a client implements the interface, we cannot change it in the future, since their code will fail to “link” at runtime due to missing method implementations. Thus, an interface in API is generally frozen forever, and must be extended in the ugly IInterface2, IInterface3 manner.

Instead, provide an Abstract... class or an ...Adapter class implementing the interface in a minimal way, which clients can use instead, and mark the interface as @noimplement.

Javadoc

Ensure that Javadoc is available on anything marked API, including description, @param tags, @return tags, @throw tags. Mark cases where null is allowed or not allowed / returned or not returned. Add @since 2.0 (2.1, etc) tags when adding new content to previously-published API.

Class naming

For consistency, use these patterns:

Base classes which partially implement an interface should be named "Abstract..." (rather than the previous Base... or ...Base pattern).

Enumerations

Do not use these in API when there is any chance that the enumeration members will ever be dynamically determined. You can't create new enumerations dynamically. Thus any API using the enumeration will have to be converted to a class, breaking compatibility.

Deprecating API

If API is to be removed, do not immediately delete it. Mark it @deprecated and keep it through the next major release.

Adding New Plugins or Features

Setup

First, ask if it is necessary to make a new one :) Certainly it's not forbidden, but it's always good to ask first.

New plugins should follow the same naming convention as other plugins (“org.maemo.<project>.<...>”).

Set up the manifest:

Add the license files (about.html, about_files/ for plugins, license.html, licenses/ for features). You can copy these from another plugin but delete the about_files/.svn folder! Double-check this carefully or else you'll get the repository into a troubled state.

Sanity checks

When you're about to finish a new plugin, check these items:

Adding the Plugin or Feature to the Build

When you're about to finish a new plugin or feature, make sure it will be built and distributed properly:

Committing a new project

Be sure the new project is inside a checked-out tree, and do:

$ svn add <project>

Then, be sure no bin/ folder is checked in:

$ svn propset svn:ignore bin <project directory>

And ensure that bin is no longer in the add queue:

$ svn revert <project>/bin

Then commit:

$ svn commit

Finishing a Feature or Bugfix

When coding is done on your branch, your feature or bugfix isn't necessarily done. It won't be in the real product until it's tested, visible to the product, and merged.

Documentation

When adding a significant new feature, either write help or API documentation immediately, or file a bug to remind someone else to do it. Or, update existing help documentation when needed.

When editing help, please use a modern HTML editor (maybe Eclipse Webtools) which can validate syntax and use CSS in previews. Avoid programs like Word or OpenOffice unless you are sure to use them in a way such that won't inject a lot of meaningless style="..." tags, since this makes it very difficult to read/edit the documentation in text.

With UI, be sure to add a help ID (WorkbenchUtils#setHelpContextId) to provide help links. These don't need to be hooked up right away, but be sure to add the code as soon as possible.

As far as API goes, something not documented will not be used or understood.

For UI, take screenshots in Linux using the Human theme. Try “gnome-screenshot -w -d 2”. If you're referencing ESbox-specific or non-Linux-hosted-specific features; use Windows XP with the default XP theme.

The main reason for making the documentation part of the finishing task is to avoid doing all this work right before the release :)

Unit tests

On your branch, follow this checklist:

It's good practice to get in the habit of running these before any commit to the trunk – you can never be sure you didn't miss something.

Validating the product

If you make changes to Mica, be sure to run both ESbox and PluThon products to make sure any new UI either shows up in both products or only in the product in which it will work. Be sure to use Window > Reset Perspective to validate you're not using cached data from the previous product launch.

If you've added new dependencies to a plugin, be sure you do not add cross dependencies between features. Keep the dependency order of features in this order:

"org.maemo.mica.common"
"org.maemo.mica.linux"
"org.maemo.mica.cpp"
"org.maemo.mica.python"
"org.maemo.mica.protocol"
"org.maemo.mica.protocol.rse"
"org.maemo.mica.maemosdk"
"org.maemo.mica.linux.autotools"
(FOR ESBOX)
"org.maemo.esbox"
"org.maemo.esbox.scratchbox"
(FOR PLUTHON)
"org.maemo.pluthon.core_feature"
"org.maemo.pluthon.python_feature"


For example, feature "org.maemo.mica.maemosdk" can't depend on "org.maemo.mica.cpp"... the dependency order is always descendent. Another important point: do not change the order of features of files feature.xml (on org.maemo.esbox-feature and org.maemo.pluthon-feature).

You can verify that the dependencies have not been broken by invoking the automated build; unfortunately this kind of “inter-feature dependency” is not detected by PDE builds inside Eclipse.

Merging to trunk

The agile development document mentioned at the top says that you “copy” changes to the trunk when done. “Copying” isn't really allowed in this sense in Subversion, so you're really just merging again. You're supposed to be synchronized with the trunk already, except for your changes, so this should be easy.

Note that, unlike the daily merge, you are applying the entire history of changes on your branch to the trunk. This is the history since you created the branch or last copied to the trunk. It's not the last time you merged from the trunk.

EXAMPLE:

Find or remember the point where you last merged your branch to the trunk. (It may not have been you who did this.) Do "svn log" inside the trunk to find the latest merge commit message.

  trunk$ svn log
  ...
  r426
  Merged revs 405:425 from your-branch to trunk
  ...

Here, 426 is the first revision to use. (Yes, use the merge commit-point itself.)

OR, if you're not reusing a branch, you may have created it and never copied it to the trunk. In this case, do "svn log --stop-on-copy" inside your branch to find out where you created it:

  your-branch$ svn log --stop-on-copy
  ...
  r400
  Created your-branch from trunk rev 399

For this case, use rev 400.

Then, check out or update a working copy of the trunk. Merge the changes, swapping the working copy you use and the branch URL you specify:

  trunk$ svn update
  At revision 510.
  trunk$ svn merge -r426:510  https://garage.maemo.org/svn/<project>/branches/<your-branch>
  # ... or -r400:510 in the alternate case above
  A ...
  M ...
  D ...

Testing merged trunk

After merging to trunk, it's best practice to repeat the steps above – open Eclipse, make sure everything still builds, then run the tests and validate the product.

You're supposed to be merged with changes from the trunk already, and you should have found and fixed any issues before your final commit to your branch, but there may be surprises.

Committing to trunk

When committing to trunk, use a commit message that reminds you (and others) what branch the changes came from.

  trunk$ svn commit -m "Merged revs 426:510 from your-branch to trunk."
  ...
  Committed revision 511.

It's not necessary to repeat all the commit messages from the branch, since everyone can look them up from the range of revisions specified. If there's some important change, though, that would be good to mention :)

Bug and feature databases

Which to use

Currently these are in the Tracker / Bug and Feature databases in the garage.maemo.org projects.

Use the Mica project trackers for issues which you know to occur in Mica (source) projects, or, if you do not know the distinction yet, for features which would appear in both ESbox and PluThon. Generally, stuff involving scratchbox is ESbox-only.

We don't use the feature database much – using product backlogs in wikis instead – but this may change in the future.

Also, we hope to use a better bug database in the future (such as bugs.maemo.org) but this hasn't happened yet. In any case, the process below mirrors the kinds of fields and terms used in Bugzilla, so it should stay the same.

Bug cycle