Time for another update on my endeavors - since last post, the change tracking application grabbed my full attention. It helps to have at least one potential "customer" around, available to bounce ideas off and ask all sorts of silly questions. And in all of my humbleness, that customer is me.

The main scenario I'm keeping in mind is the following: I have a production server running either a pre-production / production site, and a separate development instance I'm working on. I want it to be really easy to change a few things, add a feature or fix an annoying bug, and quickly update production server with that.

My current take on a solution: change tracking / uploading application. The flow:

1. [optional] Start tracking changes
2. Do the work, add a feature, close the bug, rewrite that code that suddenly doesn't look as beautiful, you know, things that developers do. Some items might get changed, new items are added. Finish, run the tests or whatever personal QA you do.
3. Open change tracker and see the changes done since you've started the change tracking, item or files-that-matter wise. If you haven't, manually select date range, like all changes done today. Possibly discard some of the changes as irrelevant. Then use the ones, that matter:
   * Upload directly to the Sitecore server.
   * Add the changes to existing or new package project. This can be used to accumulate the changes, or to have the ability to use package designer to manually retouch the changes before they go further: add   something important, for example, like your latest photo.

How it looks at the moment:

As I mentioned in the last post, it's not currently possible to replicate all types of changes using packager - i.e. you cannot tell package project to delete an item when the package is installed. That's just the way it works for now - only created and updated changes are supported. If it proves to be useful (even if only to that one customer) - I'll have more time and inspiration to enhance it.

That's also why I'm not trying to do everything at once: currently there's no intention to solve concurrency or versioning issues, with anything beyond standard Sitecore means. I just assume that you are either a single active developer on the project at the moment, or this is being handled by some other means. Hacking it is very tempting, but the ideas I came with quickly started to transform into writing something almost of source control management scale. Since I'm only working on it in my personal, spare time and being the only developer I really need to keep things simple (and fun); that's all my unfinished projects talking to me from the repositories in "inactive" folder, the dark, grim and boring graveyard of theirs.

* Previous post: Communication Between Sitecore Servers

I've ended last post saying that I'd like to be able to send packages over the wire to other Sitecore servers. Starting last weekend, it is officially cold here in Ukraine. The snow broke on Friday, and now we're steady below zero. Not much fun outside, but I had a chance to explore the communication idea a bit more.

Making Sitecore know it's neighbors.

Production Sitecore server is very rarely alone: starting from partner developers having individual instances, to build servers, QA, etc. These servers, however, do not know anything about each other.

List of known servers, or endpoints, is stored in core database. Endpoint is basically an url to Sitecore instance along with credentials matching the user account in that instance. It's not required to store username and/or password - you can type them in when they are needed instead.

Sending Data.

Now, when Sitecore knows some addresses, we can start sending gift wrapped packages. Right click on item -> Copying -> Send To -> Select one of the servers:

Then choose whether you want to send sub items or not, input credentials if they are needed and click Next. Behind the scenes, the items are placed in a package, which is then sent to a target Sitecore instance and installed there.

The usefulness of this depends on your 'process', I suppose. Working on our sites, I often have to make incremental changes and upload them to production, both before and after the site launch. When the change is really small, the burden of building / uploading / installing package is heavy enough to try to avoid adding items altogether, if possible. I'd love to hear how it works for others and what are the repetitive bits you'd like to avoid.

Change Tracking.

This part is more like what I'm looking at next, as it's very incomplete. The idea is to use history of item changes to allow developer to bundle all items she changed today, for example, and send them away in package, much like sending individual items or branches in previous part. Some prototype bits:

A bit of impedance mismatch is that it's not possible to package item deletion, i.e there's no 'anti-item' to place in package so that it deletes a certain item during installation. I enjoy using packages as communication protocol though and extending packager is an option.

Previous posts:

* Automated Package Operations
* Sitecore Build Automation

After the build automation post I felt like coding and decided to pick the low hanging fruit of package operations, that is being able to automatically install a package and build a package from package project.

Here's the current design:

1. Package installation
/Package_Install folder inside Sitecore installation is being watched for new files. Once the package file is dropped there, Sitecore automatically installs the package. Then the maintenance stuff like moving the file out and reporting is performed.

2. Package generation
Similar to installation, /Package_Generate is being watched for packager project files. Once the project file is dropped in the folder, Installer generates a package and places it near the project file.

The best part is that it almost works. Package generation works reasonably well, while instalation still requires a "don't-ask" package, installs it and then throws an exception - no big deal, we'll fix it. I'd like to add real webservices in future and be able to send packages over the wire, but this local version seems easier to use both manually and in automation scenarios.

One of the things I really want to be able to do with Sitecore is build automation. It might not be as pressing when you have one or two projects in development, but here in Sitecore Ukraine we need to constantly update and maintain a number of modules which easily exceed a number of programmers. The usual build stuff like updating and checking in the package can get pretty tedious.

The good news is that we're getting much closer to this goal with 5.3, and the new packager is a big part of this. So what do we need to make the build magic happen?

1. Building the project package(s) automatically.

It's were the new packager really rocks. Separating package definition (package project) from the actual data file is a much needed change. One of the killer features of this is being able to dynamically (conditionally) include items in the package. For instance the package project could say that all children of /system/modules/my_module and /system/templates/my_module should be included in the package when it is built - no need to update the definition each time you decide to add a new template field.

What is missing: there's still no readymade way to automatically build the package. The packager API makes it really simple though, and it would be a trivial task to add a web service that builds the package from package project file.

2. Automated package / module installation.

First, there needs to be a way to automatically install the package. The hard part then is being able to automate configuration changes as most of the settings are still stored in configuration files while packager speaks items, not xml. 5.3 makes is somewhat easier with _sc.include_. Dealing with external module dependencies can be even harder, but that's mostly out of scope of Sitecore and can be dealt with on individual basis.

What can be done: Again, packager provides API that can be used to issue 'install package x' command, so it's not hard to have this available via web service of some sort. At the moment it also involves designing a package in a way that ensures that no 'overwrite' confirmations are required - packager needs to be enhanced with 'quiet install' mode.

Automated configuration changes is still an untouched territory. Truly automated module installations is a major (and very noble) task involving lots of changes, but for the sake automated builds some shortcuts could be made. Basically you need a way of adding your custom bits to default web.config, but not sure if the half-solution is worth it though.

3. Running unit tests.

Once the package is built and installed into a prototype Sitecore instance, it's testing time. To recap: we use a custom NUnit test runner (which is simply a web page) to run NUnit tests against Sitecore in a fully setup web environment. A number of helper methods such as programmatically setting up the Sitecore site support the test framework. he custom test runner is already able to execute all tests in the supplied assembly and save the standard NUnit test report .xml to disk, which then can be analyzed to decide if the build is successful or broken. 
 

So yes, we're getting closer but not quite there yet. A lot can be built on top of what we're already have in 5.3, and apart from that I'd really like the configuration issue to be tackled. Another thing we all can benefit from is known conventions of how you do develop with Sitecore, making it easier to support it using the automation.