File sync problem and possible solutions

It is well-known fact that an overwhelming majority of people are quite lazy, especially programmers. The latter invented functions and subroutines, classes and components in order not to code things twice. In this very case it seems that a sort of creative laziness can bring huge benefits to the world we are living in. But what if a person is not lazy enough to invent a brilliant idea or use an existing one? I was that person for last several years.

My problem was connected to my work. Not to the work itself but to the fact that I usually work at the office as well as at home and I want to get the whole working environment with me wherever I go. By the environment I mean not only projects that I’m working on (this case is handled rather well with the help of hg or subversion) but the entire set of files, documents, may be some music, electronic books, etc.

I tried a number of ways. The most long-lived one was an approach with a flash drive usage. I had to plug-in it every time I came to the office and upload the updated files onto my computer. Before I left the office I had to do the reverse operation – copy updated files back to the flash drive. The same routine was waiting for me at home. If I forget any of these operations – I get out of sync and simply can’t continue my normal working process.

The other options that I tried:

• Working from home via VPN + Remote desktop. This is not very useful especially when your ISP is not stable. It requires that both computers are on, network speed is rather high.
• Using a FTP server in order to  sync 2 computers. Not convenient, counts on your manual actions. But doesn’t require that both computers are on.
• Using one of Opera unite applications such as File Sharing, or Document Sync + Document Courier bundle. The latter one is very promising though. Opera-based solutions don’t require external ftp server although the maximum size of synchronizable directory in case of DS+DC must not exceed 350 Mb.
• Microsoft FolderShare – the same as the previous one but from Microsoft. Requires a specified application to be installed. Files are stored on your computer but the application provide access to it from the other ones. So again, both computers must be on in order to access shared files.

During those unsuccessful attempts to get a bit of happiness with all that stuff I suddenly realized what I actually wanted. It was simple and genius at the same time: all I wanted is to have on both computers a folder that can automatically synchronizes with some external internet storage. No more.

So my next step was to find an internet service of such kind. But no luck. Actually, there are tons of internet services which provides you with backup capabilities of any sort but not with auto synchronization. Moreover, I started to think that there is no such service and I had to invent my own one. 
Finally, after fruitless searches I found one, but it was worth hours of googling. In fact, even google couldn’t help me as I found the answer in Wikipedia =).

So the solution is – Dropbox! It does exactly what I want, simply syncs a folder in you computer with a cloud, so there is no any necessity to keep your computer working after the sync procedure is completed. It listens to the file system events and uploads updated files immediately. You don’t have to do anything special, just install, configure and that’s it! I really loved the service. It helps to forget about sync problem and concentrate on something really important. It provides free of charge account with 2 Gb limit and it seems that it is enough for me, at least for now. There is a small video presentation on the main page, so I suppose that it is worth watching.

Don’t get me wrong, this is not an advertisement of any kind.  I decided to write the post just because I really, really impressed by the finding. Hope you’ll like it too. They say – “Happy dropboxing!” =).

A code quality assessment concept

Alex Ilyin shares an interesting concept of code quality assessment. Don’t miss it. The idea is quite fresh and promising.
The only minus that I can see is that the actual realization might be a bit time consuming.

Ayende on Dependency Injection

Here is another post from Ayende, this time on rejecting Dependency Injection inversion. It is worth reading and highly recommended for those, who don’t like the very idea of IoC usage. For example, Alex Ilyin is from that camp. Recently we had quite interesting conversation on this topic in comments to this post.

Updates on the development process

The beginning of 2010 year was quite busy for the entire DataObjects.Net 4 team. One part of it headed with Alex Yakunin, has been busy with the late DataObjects.Net 4.1 release (which in fact still haven’t been shipped), the other has been working hard on the upcoming DataObjects.Net 4.2 release.

While other developers from the latter part have been working on full-text search, object-to-object mapper and other features from 4.2, personally I was working on highly demanded by customers SQL Server Compact Edition 3.5 support. The work is already done, CE support related code was committed and is being tested by our CI server during the last 2 days. The feature is going to be included into 4.2 release. As for Sql Server CE, it has a huge number of restrictions and I’m going to list all of them in one of the future posts. To sum up, it can replace its big brother (Sql Server 2005/2008) only partially.

Speaking about the upcoming DO.Net 4.2 release, we are planning to make a stable Release Candidate at the end of the January (at the beginning of the February) and deliver DO.Net 4.2 final in the middle of the February, so stay tuned.

As for me, I’ve finished working on 4.2 branch and there is a high possibility that I’ll be busy during the next 2 or 3 months with another project, not connected with DataObjects.Net development at all. The project is going to be very complicated and, as it usually happens, has very limited time-frame, but I hope that my participation will help a lot/a bit. My role will be mainly as business analyst/system architecture. I’m preparing myself for working on endless Word documents, requirements, UML diagrams, precedents, etc. So wish me good luck. =)

However, if I have enough time I’ll keep on posting some regular notes on DO.Net development process.

A frank confession

Alex Ilyin lifts the curtain on terrifying details of DataObjects.Net 4 development process.

Query preprocessors, Inversion of control & Localization support

LINQ translator extension

As I promised earlier, we were going to make LINQ translator extendable and finally we’ve made this. The extension mechanism is called “Custom LINQ query preprocessors” and is already included into DataObjects.Net 4.1 code base.

Custom LINQ preprocessors must implement public interface IQueryPreProcessor which is defined in Xtensive.Storage assembly. Here it is:

public interface IQueryPreProcessor
{
  Expression Apply(Expression query);
}

As you might see, the contract is quite simple and straightforward: your preprocessor receives the whole query, modifies it in  the way you need, and returns the modified one. All preprocessors are called before the query is processed by internal LINQ translator, so it is the right time and place to apply necessary modifications.

Connecting preprocessors to translator (IoC)

After you have written you preprocessors, it is time to plug-in them to DataObjects.Net. This is done with the help of Inversion of Control concept. In order to follow it, you need to take the following steps:

1. Add reference to Microsoft.Practices.ServiceLocation.dll assembly. It is shipped with DataObjects.Net 4 and can be found in %DataObjects.Net Directory%\Lib\CommonServiceLocator directory.

2. Configure IoC container through application configuration file.

Add this line to configSections part:

    <section name="Services" type="Xtensive.Core.IoC.Configuration.ConfigurationSection, Xtensive.Core"/>

Add the corresponding configuration section:

<Services>
  <containers>
    <container name="domain">
      <types>
        <type type="Xtensive.Storage.IQueryPreProcessor, Xtensive.Storage" mapTo="Xtensive.Storage.Samples.Localization.QueryPreProcessor, Xtensive.Storage.Samples.Localization" singleton="true" />
      </types>
    </container>
  </containers>
</Services>

Note the usage of named service container (“domain”), the  IQueryPreProcessor type as an interface of a service and how it is mapped to the concrete implementation.

3. The last step is to configure Domain object & the above-mentioned service container.

// Building domain
domain = Domain.Build(DomainConfiguration.Load("Default"));

// Configuring domain-level services
var configurationSection = (ConfigurationSection)ConfigurationManager.GetSection("Services");
var container = new ServiceContainer();
container.Configure(configurationSection.Containers["domain"]);
domain.Services.SetLocatorProvider(() => new ServiceLocatorAdapter(container));

LINQ preprocessor in action

Having these actions done, we get the capability of using non-persistent localizable properties (Domain model can be found here) in LINQ queries:

using (var ts = Transaction.Open()) {

  Console.WriteLine("Implicit join through preprocessor");
  var pages = from p in Storage.Query.All<Page>()
  where p.Title=="Welcome!"
  select p;
  Console.WriteLine(pages.ToList().Count);

  ts.Complete();
}

Pay attention that neither PageLocalization type nor its members participate in the query, original p.Title expression in Where clause is used instead. As we know, Page.Title is not a persistent property and regular LINQ translator doesn’t know how to translate this expression. But having the initial query preprocessed with Xtensive.Storage.Samples.Localization.QueryPreProcessor makes such kind of expressions possible to use. The only thing the preprocessor makes is the replacement of p.Title expression to something like this:

p.Localizations.Where(localization => localization.CultureName==LocalizationContext.Current.CultureName)
          .Select(localization => (string)localization[“Title”])
          .FirstOrDefault();

That’s it.

The source code is available in our public repository in Xtensive.Storage.Samples.Localization folder.

Happy preprocessing! =)

Logging, part 3. Configuring logging through log4net

In the previous post I demonstrated how to configure and use internal DataObjects.Net logging capabilities. These are rather useful and flexible but in case you want much more flexibility or something, using one of the external logging frameworks could be the right choice for you.

As I already mentioned, external logging components are connected to DataObjects.Net through the mechanism of adapters: DataObjects.Net => adapter for logger => logger.

In case of log4net you’ll need the following assemblies:

• log4net.dll (can be found here)
• Xtensive.Adapters.log4net.dll (is contained in DataObjects.Net installer)

The next step is to configure both DataObjects.Net & log4net.

Add these sections to configSection block of your application configuration file:

    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net"/>
    <section name="Xtensive.Core.IoC" type="Xtensive.Core.IoC.Configuration.ConfigurationSection, Xtensive.Core"/>

Xtensive.Core.IoC namespace goes for basic Inversion of Control implementation however it is powerful enough to accomplish most of appropriate tasks. In this case it is used to map Xtensive.Core.Diagnostics.ILogProvider interface to some external implementation (Xtensive.Adapters.log4net.LogProviderImplementation type).

<Xtensive.Core.IoC>
  <containers>
    <container>
      <types>
        <type type="Xtensive.Core.Diagnostics.ILogProvider, Xtensive.Core" mapTo="Xtensive.Adapters.log4net.LogProviderImplementation, Xtensive.Adapters.log4net" singleton="true"/>
      </types>
    </container>
  </containers>
</Xtensive.Core.IoC>

The last step is log4net configuration:

<log4net>
  <appender name="FileAppender" type="log4net.Appender.FileAppender">
    <file value="log-file.txt" />
    <appendToFile value="true" />
    <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date %-5level %logger - %message%newline" />
    </layout>
  </appender>
  <root>
    <level value="WARN" />
    <appender-ref ref="FileAppender" />
  </root>
  <!-- To log warnings & errors from Xtensive.Storage.* loggers --> 
  <logger name="Storage" additivity="false">
    <level value="WARN" />
    <appender-ref ref="FileAppender" />
  </logger>
  <!-- To log all SQL statements -->
  <logger name="Storage.Providers.Sql" additivity="false">
    <level value="ALL" />
    <appender-ref ref="FileAppender" />
  </logger>
</log4net>

Having these configuration steps done, you’ll get DataObjects.Net & log4net bundle configured & working.