Friday, October 30, 2009

DataObjects.Net v4 manual

Finally, we started with long awaited DataObjects.Net v4 full-fledged manual.

But before we started we had spent lot of time in discussions about tools to be used for manual creating.
Three options were suggested:

Microsoft Word approach

Use Microsoft Word for editing then export as HTML and clean up the resulting HTML from Word-ish garbage.

  • Easy editing & formatting;
  • Proofing;
  • Inserts images;
  • Inserts highlighted source code from Visual Studio.
  • You must run “Export to HTML” & make a cleanup every time you want to view page in browser;
  • Almost no version control;
  • Difficult to make references between resulting HTML pages;
  • Difficult to manage inserted images.

Wiki approach

Use Wiki to build the whole manual then grab its content and convert into local HTML pages.

  • No need for local editors, just your browser and you;
  • Simple version control;
  • Supports highlighting of source code fragments.
  • Grabbing & converting wiki into manual is not a trivial task;
  • No proofing (proofing can be provided by some browsers while writing text in textarea control);
  • Wiki markup (you are to know it to format your text).

Plain HTML approach

  • There are a lot of WYSIWYG HTML editors that support Microsoft Word’s functionality (formatting, proofing);
  • Clean HTML code (you control it);
  • You see the resulting HTML at any moment (no need for grabbing or conversion);
  • You control where and how your images are placed and named;
  • Making references between pages is simple;
  • Full version control support as HTML file is a text file.
  • Good enough HTML editor must be found;
  • No support for inserting highlighted source code fragments from Visual Studio.


After intense discussions we decided to use Plain HTML approach. Firstly we used SharePoint Designer 2007 as HTML editor, then moved to Visual Studio. At last we found Microsoft Expression Web 3, which actually had been made on top of SharePoint Designer. It is faster than VS and more convenient. I like it except its black theme – it is too dark for me.

Microsoft Expression Web 3

One more useful tool that we found is “Copy As HTML” add-in for Visual Studio. It allows you to copy source code from the Code Window and convert it into HTML while preserving syntax highlighting, indentation and background color.

Copy As HTML


And what about you? What tools do you use to write help or manuals? Are there any other, more powerful tools or more convenient ways to do this?

Thanks in advance for your ideas.


Tuesday, October 27, 2009

Arbitrary keys & hierarchies. Complete reference set

Here are references to all parts of the “Arbitrary keys & hierarchies” series gathered in one place:

  1. Introduction
  2. Hierarchies
  3. Evolution of Key
  4. Working with keys
  5. Key providers
  6. Identity fields
  7. Custom key generators

Thanks for your interest in DataObjects.Net v4.


Monday, October 26, 2009

Arbitrary keys & hierarchies, part 7. Custom key generators

As DataObjects.Net v4 supports wide variety of keys but has default implementation only for key generators with exactly one identity field, there could be scenario in which custom key generator is required. To close the gap DataObjects.Net v4 declares the following abstract class:

public abstract class KeyGenerator
  public KeyProviderInfo KeyProviderInfo { get; private set; }

  public abstract Tuple Next();

  public virtual void Initialize() {}

  protected KeyGenerator(KeyProviderInfo keyProviderInfo) {}

Custom generator type must inherit KeyGenerator type and implement at least the abstract method KeyGenerator.Next(). All necessary information concerning the structure of key, caching behavior and so on can be found in KeyProviderInfo type. Here is the custom implementation of key generator for Guid type:

public sealed class MyGuidKeyGenerator : KeyGenerator
  public override Tuple Next()
    return Tuple.Create(KeyProviderInfo.TupleDescriptor, Guid.NewGuid());

  public MyGuidKeyGenerator(KeyProviderInfo keyProviderInfo)
    : base(keyProviderInfo)

In order to indicate that a hierarchy must be served with custom key generator, KeyGeneratorAttribute was introduced. Here is how it is intended to be  used:

public class Author : Entity
  [Field, Key]
  public Guid Id { get; private set; }

  public EntitySet<Book> Books { get; private set; }

In scenarios when key generator for a hierarchy is not required at all, this must be set up appropriately:

[Index("Name", Unique = true)]
public class Type : MetadataBase
  [Field, Key]
  public int Id { get; private set; }

  [Field(Length = 1000)]
  public string Name { get; set; }

  public Type(int id, string name) 
    : base(id)
    Name = name;

Here is how system class Metadata.Type is declared in DataObjects.Net v4. Pay attention to KeyGeneratorAttribute usage together with the absence on parameterless constructor. This means that identity field values are provided from the outside and there is actually no need in key generator.

This is the last post in “Arbitrary keys & hierarchies” series. Hope you’'ll find it useful.

Part 6. Identity fields

If you want me to blog on some particular topic concerning DataObjects.Net v4 domain – make a request in comments.


Arbitrary keys & hierarchies, part 6. Identity fields

DataObjects.Net v4 supports the following .Net types to be used in identity fields:

  • Boolean;
  • Byte, SByte, Int16, UInt16, Int32, UInt32, Int64, UInt64;
  • String, Char;
  • Double, Single, Decimal;
  • Guid, DateTime, TimeSpan;
  • Reference to Entity (is stored as Key)

Usage of Structure and EntitySet<T> types is not allowed, however restriction for Structure usage could disappear in future versions of DataObjects.Net.

Identity fields are set once for all persistent hierarchy, so all descendants of hierarchy root share the same Key structure. In order to set up structure of Key KeyAttribute class should be used:

public class Animal : Entity
  [Field, Key]
  public int ID { get; private set; }

  public int Name { get; set; }

Key attribute must be placed on each identity field. Also note, that identity field must be immutable, it is prohibited for identity field to have a public, protected or internal setter.

For persistent types with complex keys (which have more than one identity field) explicit identity field order is strictly recommended, because the particular order of list of properties, got with the help of .NET reflection, is not guaranteed at all.

public class BookReview : Entity
  [Field, Key(1)]
  public Person Reviewer { get; private set; }

  [Field, Key(0)]
  public Book Book { get; private set; }

  [Field(Length = 4096)]
  public string Text { get; set; }

  public BookReview(Book book, Person reviewer)
    : base(book, reviewer)

Pay attention to KeyAttribute usage (Key(0), Key(1) lines). In this example it also used to set the position of identity field within complex key. So for BookReview type the structure of Key is {Book, Person}. Also note that values for both identity fields are required in BookReview constructor and passed to the base constructor of Entity where key is constructed.

Part 5. Key providers, Part 7. Custom key generators


Friday, October 23, 2009

Arbitrary keys & hierarchies, part 5. Key providers

In the previous posts we discussed the evolution of Key, its structure and the ways to work with keys. Now it’s time to answer the question: where keys come from? But first, let’s see scenarios where we do need new keys.

As you know, Entity instance is uniquely identified by key and Entity.Key property is immutable during whole Entity’s lifecycle. The only moment when Entity.Key property can be set is the moment of Entity construction. There are two scenarios and DataObjects.Net v4 supports both of them:

1. Values of identity fields are provided by outer code (not ORM but application is responsible for this). In this scenario user code is responsible for passing identity values directly to Entity constructor where these values are automatically transformed into key.

// Use this constructor for types with explicitly set identity values
protected Entity(params object[] values)


public class Book : Entity
  [Field, Key]
  public string ISBN { get; private set; }

  // Accepts identity value (ISBN) and passes it to the base Entity constructor
  public Book(string isbn)
    : base(isbn) { }

2. Values of identity fields are provided by ORM (or with the help of ORM). This scenario might involve usage of database identity generators, tables with auto-increment column as well as any custom identity generators, e.g. Guid generator in order to get identity values for an Entity and build its key.

// Use this constructor for types with auto-generated identity values
protected Entity()


public class Book : Entity
  [Field, Key]
  public int Id { get; private set; }

  // Nothing to be done here as base empty Entity constructor will be called automatically
  public Book() { }

While the first scenario is clear and doesn’t need any ORM participation at all (except creating key from values passed to constructor), the second one is not so clear because it requires ORM to know how to obtain the next unique identity values for the specified persistent type. To solve this task the concept of key providers was introduced in DataObjects.Net v4. KeyProviderInfo is a type from Domain model which contains all necessary information concerning generation of unique keys for particular persistent type(s), including:

  • type of identity generator implementation;
  • structure of key (key columns and descriptor of tuple);
  • size of key cache, if generator supports caching;
  • mapping name, if generator is mapped to a particular database entity;
  • etc.

DataObjects.Net v4 contains several key generator implementations both for In-Memory & SQL storages out of the box. They are called default key generators. They support caching and can be used for persistent types with exactly one identity field. This means that for persistent types with complex key (more than one identity field) custom key generator must be provided by user. In order to increase key generator performance and to simplify persistent interfaces support we made the following architectural decision:

We try to share single instance of every key generator type between all hierarchies it can serve.

All hierarchies which have identical key structure and identical key generator type, e.g. typeof(MyKeyGenerator), are served by the same instance of MyKeyGenerator. In other words, we create only one instance of each key generator type, registered in Domain. For example:

public class Book : Entity
  [Field, Key]
  public int Id { get; private set; }

  public Book() { }

public class Author : Entity
  [Field, Key]
  public int Id { get; private set; }

  public Author() { }

Here we have 2 hierarchies with default key generator (default key generator is used unless custom key generator is explicitly defined via KeyGeneratorAttribute). As both hierarchies have identical key structure (one field type of int) and the same type of key generator – default key generator, then only 1 key generator (Int32-Generator actually) will be created and it will serve both hierarchies at once. This means that sequence of Book identifiers as well as sequence Author identifiers won’t be strictly sequential. Say, one can get 1,2,3,6 for Book identifiers and 4,5,7,8,9 for Author identifiers. Nevertheless, all keys produced by this key generator are unique and can be used in any number of hierarchies which can be served by this key generator type without any restrictions.
The only one “negative effect” of this scheme is the presence of gaps in identifier sequence for concrete persistent type. But does it matter? Such gaps are common as objects don’t live forever, sometimes they are removed.

In the next post I’ll tell you about complex keys and custom key generators and their future.

Part 4. Working with keys, Part 6. Identity fields


Wednesday, October 21, 2009

Arbitrary keys & hierarchies, part 4. Working with keys

In the previous post I described the structure of Key. Now it’s time to discuss the ways of working with keys.

Obtaining a key

Key can be obtained from Entity through Entity.Key property, even if Entity instance marked for removal (Entity.IsRemoved == true).

Creating a key

There are 2 common scenarios in which instance of Key need to be created:

1. New Entity instance is created via its constructor. In this scenario we need to get the next unique key in corresponding key sequence for the specified persistent type. This can be done by these group of static methods:

// Active session is required. It is used to get access to Domain object.
Key.Create(typeof (Dog));

// Active session is not required as Domain is passed as argument.
Key.Create(Domain, typeof (Dog));

Note that methods don’t receive any arguments except Domain & Type. This means that we ask to generate the next unique key in key sequence.

2. For example, to fetch Entity instance from storage we need a key that identifies it. In this case we need to construct a key with already known value(s).

// Active session is required
Key.Create<Dog>(params object[] values);
Key.Create(typeof (Dog), params object[] values);

// Active session is not required
Key.Create<Dog>(Domain, params object[] values);
Key.Create(Domain, typeof (Dog), params object[] values);

If we want to build a key for an instance of Dog class with identifier equals to 25 we write something like this:

var key = Key.Create<Dog>(25);

Or we can use one of three other overloads.

Another group of methods for building a key accepts instance of Tuple.

// Active session is required
Key.Create<Dog>(Tuple value);
Key.Create(typeof (Dog), Tuple value);

// Active session is not required
Key.Create<Dog>(Domain, Tuple value);
Key.Create(Domain, typeof (Dog), Tuple value);

Accordingly, to build key from Tuple for an instance of Dog class with identifier equals to 25 you write:

var value = Tuple.Create(25);
var key = Key.Create<Dog>(value);

As you see, there is no public constructor in Key class. The reason for this and the usage of Factory method pattern instead of constructors is that we have several Key implementations, 4 of them are designed and extremely optimized for short keys (Key<T1>, Key<T1,T2>, Key<T1,T2,T3>, Key<T1,T2,T3,T4>) and one is for keys with unrestricted length (LongKey).


Serializing & deserializing a key

Key can be easily serialized into the corresponding string representation using Key.Format() and deserialized from the string using Key.Parse method.

For example:

var key = Key.Create<Dog>(25);
var str = key.Format();
// This will print: "103:25"
// where 103 is identifier of Dog type in Domain.Model
// and 25 is the value of identifier field.

var key2 = Key.Parse(Domain, str);
Assert.AreEqual(key, key2);

That’s all about working with keys. In the next posts I’ll describe key providers and key mappings. Stay tuned.

Part 3. Evolution of Key, Part 5. Key providers


Monday, October 19, 2009

Arbitrary keys & hierarchies, part 3. Evolution of Key

As you might remember, Int64 type was used as a global unique identifier in DataObjects.Net v3. What were the reasons for changing this working approach? I’ll tell you: the main reason was the strong requirement to support Bottom-up development model. In case when a customer has a database and he wants to use an ORM to work with, in that case we just can’t say: “Hey, man, change all primary keys in all your tables to bigint (analogue of Int64 in Microsoft SQL Server) in order to use our super-duper ORM!”.
So if we want to support existing databases, we must support all kinds of primary keys that can potentially be implemented there.

How could we do it?

First of all, let’s enumerate possible logical structures of primary key:

  • single value key
  • multiple value key

Also note that from physical point of view primary key can contain field(s) of various types: int, long, Guid, string and so on. If so, we need some structure that can hold one or more fields of various types: something like List<object> or object[].

Taking into consideration primary key immutability, we must also make Key immutable. So some kind of ReadOnlyList<object> wrapper must be applied on top of initial List<object>.

Thus, at this moment Key class will contain List<object> and implement some interface to expose values in read-only manner.

public class Key
  private List<object> values;

  // Safe way to expose values 
  public object[] GetValues()
    return values.ToArray();

  public Key(params object[] values)


Going further.

Should two instances of Key with equal field structure and equal values be considered as equal or not? It seems that yes, they must be equal. Then in order to meet the requirement, we must override GetHashCode & Equals methods (where we are to provide field-by-field value equality check) as well as implement IEquatable<Key> interface.

public class Key : IEquatable<Key>
  // Equality support methods 
  public bool Equals(Key other)
  public override bool Equals(object obj)
  public static bool operator ==(Key left, Key right)
  public static bool operator !=(Key left, Key right)
  public override int GetHashCode()


The next step is an attempt to implement an Identity map pattern, which will be responsible for correspondence between Keys and Entities. The implementation will be simple and straightforward: we’ll use Dictionary<Key, Entity> for it.

Imagine the following domain model:

We have 2 persistent types here: Dog & Cat. Structure of identity fields are equal: one field of int type. If both classes are mapped to separate tables with simple autoincrement identity field then there is high probability of situation when values of identity fields from Dog & Cat tables will be equal. Say, we could have the following keys: dogKey = new Key(25) for Dog instance and catKey = new Key(25) for Cat instance, where 25 is the value of identity field.

var identityMap = new Dictionary<Key, Entity>();

var dogKey = new Key(25);
var catKey = new Key(25);

var dog = Query<Dog>.Single(dogKey);
identityMap[dogKey] = dog;

Assert.IsNotNull(identityMap[dogKey]); // True
Assert.IsNull(identityMap[catKey]);    // False

Pay attention to the last line. Although we didn’t add any Cat instance to the identity map, it says that it has one, either for dogKey or catKey. The reason is that both keys are considered as equal. So the problem is that keys with equal values for the same type must be equal, but for other types mustn’t be. In order to solve the problem we must distinguish keys made for different types, i.e. inject some type-dependent information into Key and take this into account in Equals & GetHashCode methods implementation. The most evident approach is to add property of Type type.

public class Key : IEquatable<Key>
  public Type Type { get; private set; }

  public Key(Type type, params object[] values)

Now we’ve got good chances to build a fully functional identity map.

So what’s then? Let’s look at the result and calculate the cost of Key usage.

  • Every Key instance has a reference to List<object> where values of identity fields are stored. Identity field values are stored as objects, so creating a Key with 1 identity field leads to creating of 2 objects in managed heap: first is for List<object> (identity field values container), and the second is for identity field value. More identity fields in Key => more small objects in heap. Not good.
    Fortunately, we can easily substitute ineffective List<object> to quite effective and compact Tuple implementation with typed access to fields (and yes, Xtensive made its own Tuple implementation. I’m going to describe it in details in a separate post).
    This approach is highly scalable and can be used in a wide variety of scenarios: from widespread scenario with only one identity field (type of int, long, Guid, etc.) to rare ones with a group of identity fields (we call such keys as complex keys).
  • Every Key instance has a reference to Type object. But before DataObjects.Net v4 could somehow manipulate with Key instance it must get corresponding TypeInfo object for it. TypeInfo is a class from Xtensive.Storage.Model namespace that fully describes persistent type in DataObjects.Net v4 domain. Resolving TypeInfo for Type costs 1 lookup in Dictionary<Type, TypeInfo>. In order to prevent these lookups (as they are going to be rather frequent) we decided to put reference to TypeInfo directly into Key instead of reference to Type.

Here is how Key class is really designed:

public class Key : IEquatable<Key>
  public Tuple Value { get; }
  public TypeInfo Type { get; }

  // Instance methods 
  public bool Equals(Key other)
  public override bool Equals(object obj)
  public static bool operator ==(Key left, Key right)
  public static bool operator !=(Key left, Key right)
  public override int GetHashCode()
  public override string ToString()

  // Static methods 
  public static Key Parse(string source) + 1 overload
  public static Key Create() + many overloads

In the next posts I’m going to describe other topics related to Key: the absence of public constructor, serialization, patterns of creations, key generators (key providers), identity fields mappings, other speed and resource utilization improvements.

Stay tuned.

Part 2. Hierarchies, Part 4. Working with keys


Tuesday, October 13, 2009

How to detect whether application is being run in Debug mode?

I don’t mean the question “How to detect whether application is built in Debug configuration”. It is achieved with the help of conditional compilation.
I mean – how to detect whether your application is being debugged from Visual Studio or other external debugger (no matter in which configuration it is built: Debug, Release or something else).
The answer is simple:


The property returns true if any debugger is attached to the application’s process.

We use this approach in DataObjects.Net v4 internal logging subsystem: if DataObjects.Net v4 is built in Debug configuration, logging is always enabled and by default log messages are written into DebugLog (it redirects them to Debug.WriteLine method). In Release configuration we check the presence of attached debugger. If debugger is attached, we follow the same logic as in Debug configuration; otherwise all messages are written in NullLog (/dev/null). So no performance overhead is produced.

Sample code that demonstrates the approach:

    protected override ILog GetLog(string name)
      return new DebugLog(name);
      return Debugger.IsAttached ? new DebugLog(name) : new NullLog(name);

.NET Reflector 6 EAP

Another great news! .NET Reflector project is not dead, it is being intensively developed.
.NET Reflector 6 ships with Visual Studio AddIn which makes available step-through-debugging on assemblies that you don't have the source code for.

More info & download link: Red Gate's forum, Alex Davies's blog.

ReSharper 5.0 is coming!

Great news! New version of this fantastic development tool is coming. Nowadays I just can’t imagine my development process without it.

IMO, the most interesting features in upcoming release are:

  • Call Tracking (to inspect an entire call sequence)
  • Value Tracking (data flow in your project)
  • Advanced bookmarks
  • Solution-Wide Warnings and Suggestions
  • Visual Studio 2008 support (yes, it is important)

The only thing I’m concerned about is its performance.

All detailed information & screenshots can be found here (if you’ve missed the link in the first paragraph).

Sunday, October 11, 2009

97 Things Every Software Architect Should Know

The book is the collective contribution of technical architects from all over the world working across the full range of domains. I found it quite useful.

The online (unedited) version can be found here. Offline ones: O'Reilly Media,

Friday, October 2, 2009

Arbitrary keys & hierarchies, part 2. Hierarchies

In the previous part I described the reasons for Hierarchy & Key terms introduction. Today we’ll continue with this topic and see how these concepts are implemented in DataObjects.Net 4. Let’s start with Hierarchy.

Hierarchy is entirely defined by its root element. This is how we do it:

public class Animal : Entity
  [Field, Key]
  public int Id { get; private set; }
  public string Name { get; set; }

Note that the class is marked with HierarchyRoot attribute and it contains one field that is marked with Key attribute. Thus we explicitly define structure of Key for this hierarchy (and by that for every class that belongs to this hierarchy).
Also remember, that all root’s descendants belong to the same hierarchy. Class could belong to one hierarchy only at a time. This means that an attempt to define a new hierarchy root inside existing hierarchy is considered as error.

Another option, that is exposed by HierarchyRoot attribute, is inheritance mapping scheme. DataObjects.Net 4 supports the following schemes:

  • ClassTable is the default one. It represents an inheritance hierarchy of classes with one table for each class. It is ideal for deep inheritance hierarchies and queries returning base classes. This case implies joins + a single base table for the whole hierarchy.
  • SingleTable represents an inheritance hierarchy of classes as a single table that has columns for all the fields of the various classes. This kind of mapping is more preferable for tiny inheritance hierarchies, or for hierarchies where there is a set of abstract classes and a non-abstract single leaf.
  • ConcreteTable represents an inheritance hierarchy of classes with one table per concrete class in the hierarchy. This kind is ideal when you always query for _exact_ types of objects stored there. I.e. you query not for Animal, but for Dog (which is a leaf type in hierarchy). When you query for Animal here, there will be UNION in query.

More detailed description of inheritance hierarchy mappings can be found in Martin Fowler’s book “Patterns of Enterprise Application Architecture”.

This is how we can define inheritance mapping for the given hierarchy:

public class Animal : Entity


As you’ve already seen – DataObjects.Net 4 gives more freedom to the way how persistent classes can be mapped to database tables. But more freedom also means more responsibility as it is required that every persistent class from domain model must belong to some explicitly defined hierarchy. Let’s look at the following domain model:

Classes marked with orange color (Dog, Cat) are hierarchy roots, but Animal – is not. As Animal class doesn’t belong to any hierarchy, DataObjects.Net 4 can’t make any decision on how Animal type is mapped to database schema; how instances of Animal type should be persisted and fetched; and even what is the structure of key for Animal type. So should this Domain model be considered as erroneous or not? We think, no. In this case Animal class could be safely removed from Domain model because every descendant of Animal class is correctly mapped. All persistent fields that are defined in Animal class will be copied into Dog & Cat classes.

In the next model such kind of removal is impossible:

Again, classes marked with orange color (Dog, Cat) are hierarchy roots, whereas Animal & Horse are not. DataObjects.Net 4 can’t make any mappings for Horse, and moreover it can’t be safely removed because we can’t make any conclusions about Horse’s hierarchy: it doesn’t have any descendants at all. It seems that in this case domain model author simply forgot to mark Horse class with HierarchyRoot attribute. So he will get a DomainBuilderException then.

One more erroneous scenario:

One persistent class (Cat) has a reference to another persistent class (Horse), but second one doesn’t belong to any hierarchy (thus its key structure is not known), so DataObjects.Net 4 can’t choose the structure of reference field in Cat table and DomainBuilderException will be thrown.

More freedom => more responsibility.

Part 1. Introduction, Part 3. Evolution of Key.