SQLMetal woes: No sqlmetal.exe with VS 2008 Team Edition

Even despite all the performance issues I wanted to use LINQ queries for one of my pet projects. One of the design goals was to use Sql Server Compact edition 3.5 so that I get to use a file as a database and also that I can port the data access and code logic to compact framework so that it can be used with Windows Mobile devices. Using compact edition of sql server 2005 is kinda like using a comptible microsoft made firebird sql database. A few days ago I reinstalled my OS and wanted to try the Visual Studio 2008 team edition trial and that is when all the trouble started.

First, the linq to sql class generator SqlMetal.exe was nowhere to be found. It seemed that the Visual Studio 2008 Team Edition trial did not come with the SqlMetal.exe. Infact the folder 'C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\' where the executable resides with other executables is empty. But I have changes to my database schema and did not want to hand write the changes.

Then I tried generation by dragging the sql server compact dbatabase tables onto the linq to sql (dbml) designer, but got the message that "The selected object(s) use an unsupported data provider". This is annoying. Tomorrow I will try to download the Visual C# 2008 Express edition and install that to get a copy of SqlMetal.exe. I wish Microsoft were more careful with what content they have in their DVDs.


New features: .NET Compact Framework 3.5

.NET Framework for Devices has given birth to numerous Windows Mobile based Applications. I have recently been interested in the compact framework because I had bought windows mobile based HTC P3300 with a built in GPS. Previously I wrote code for compact framework 2.0. Since everything has gone 3.5 now a days, I began to wonder what is new in 3.5 and how can I take advantage of the new features. I am going to write down the my finding below.

By the way, this probably is old news to some of you ... so you might not want to continue reading.

First the bad news: No WPF

I was expecting WPF to be present in CF3.5 but there was no WPF, which is quite sad. Instead there were some extremely minor improvements to the windows forms area. However don’t be sad, because Silverlight is coming with support for Windows Mobiles and even Nokia phones.

Good News: Windows Communication Foundation is here

For service based and other communications WCF is present in 3.5 and the client services in the device can connect to the WCF service running in the PC. However the service runtime layer is not present in the compact framework. For ServiceMode contracts you will need to generate a proxy using the NetCFSvcUtil which is present in the Power Toys for .NET Compact Framework. It also supports a subset for WS-Security 1.0 specification is supported. For more on this topic please look up Messaging in the .NET Compact Framework at MSDN.

Second good news: LINQ is supported

Oh yes, … now you can write sql like strange constructs in your CF code with Linq for your device. However for XLinq XPath is not supported.

Minor improvements

If you have hardware support then the Soundplayer can play multiple sounds. See this article in MSDN for details. Unfortunately support for compressed sound is not there yet. You can get OS platform information on WinCEPlatform Enumeration class at Microsoft.WindowsCE.Forms namespace. Some other improvement were done in Logging, GAC and other areas. For more on this please read What's New in the .NET Compact Framework Version 3.5 at MSDN. You can find CF samples at this article ...NET Compact Framework Samples.

For those of you who were hoping make nice WPF apps for CF start using Silverlight.

kick it on DotNetKicks.com

Tiny Utility: Extract Sharepoint Template

Recently I needed to find out the internal contents of a few Sharepoint template file which has the extension "stp". Upon close observation it was observed that the file header started with "MSCF" which is the header for a cab file. I extracted the cab file and found an xml file called manifest.xml and a bunch of other files which looked like this.

a00000.000
b00000.000
...
...
x00000.000

When I looked closer I found that these are actually the files inside a sharepoint web site and the manifest.xml has the location where the files are located. But at the current extracted format the name and location of the files made no sense. However the mapping for these files are written in the manifest file.  So I wrote a small program that read from the manifest.xml file and moved the x0000.000 named files into appropriate folders and directories. So now we have a sharepoint template extraction utility. Please note that the manifest.xml may have a lot more file listed that was once present in your sharepoint site but those files may not present inside the template file thus is not extracted. It looked to me like the manifest file contained all garbage historical files.

To see this, please download Microsoft Cabinet SDK from from here: http://support.microsoft.com/kb/310618

Then compile this attached project and decompile your sharepoint template.

Download SharePointExtractor.zip

kick it on DotNetKicks.com

Linked In recommendations: Your job happiness Barometer

“A purely non geeky writing for geeks”

Disclaimer: The article reflects my personal view based on my small experience in life.

How do you tell if your employee is not happy and is dropping CV at different places? How can you tell that the organization you are working for is not running so well and there may be job cuts? How do you tell if your colleague is planning to move to a new job? 

Well ... there are many ways of finding that out but one of the early indicators can be that person's Linked In activity. I must say that it is not a definite pointer but it's a trend that you can watch.

Recommendations
First early indication of people wanting to change their jobs is that they will start to get recommendations. This process may start one to six months before they change their jobs but the usual time is 3 months before. Now how does this work or how is getting recommendations related to changing jobs?

Well this is how it works ... When someone is unhappy with his job and thinks or plans to move on, then he tries to get recommendations for finding a new job. This is quite rational since the person thinks that a recommendation might help him/her in the future job. Now this may not be done at the very moment the person decided to leave but at the very moment he seriously considered getting another job.

So if you are working for some company and your supervisor starts getting recommendations then you should too. Or if you are in a startup company and your CTO, CEO are getting recommendations then start packing because the company is going down and you are among the first ones to go out the door.

Check out new company before you join
If you are going to join a new company check out their employees at Linked In to see how many of them have gotten recommended in past 3 months. If the number is high then it is an indication of unhappy work environment or a job without career growth.

Other elements
Now a days people have become very expressive and they want to let the world know what they are doing by using Facebook or Twitter. But nothing tells you that someone is looking for better options like Linked In recommendations.

Why?
What I have told you sounds stupid? Can’t people go and ask for recommendations for other reasons? Some people do. But most of us don’t. And if you are a software developer and a good one then I must say that you don’t have a habit of changing jobs. So when do people keep asking for recommendations? (a) when they are angry on their supervisor , (b) when they feel that that their current job is not taking them anywhere, (c) people ask their supervisors to recommend them when they think they will look for better opportunities and it is the best time to get recommendations from their supervisors before thing get bitter. 


Strange C# 3.0 Feature: Partial Methods ... ... What ?!!

At first it was hard to believe or even think up of the concept of what might be a partial method. How can you have a partial method? How does one write it ... by omitting some code? Looks like the C# team have given birth to strange new language feature called partial methods.

Before I go any further there are 2 rules regarding partial methods that must be said.

1. Partial methods can only defined in a partial class

2. Partial method return type must be void and cannot contain any out parameter

Perhaps the naming of partial method has been wrong. It would have been better to call it skeleton method. A partial method is a method that can be called in a function and the signature of the function may be defined but may not have any body. It is like defining a method signature which we may implement later.

For example lets assume we have a class called Foo which has a method called do something and looks like code below

partial class Foo
{
    public void DoSomething(string someParam)
    {            
        // Do something important here ...some logic
        CallSomeMethodSomewhere(param);
    }
}

It is visible that the we are going to do something important inside the method. While we are writing this method we may decide that we want to log the input parameters but at the time we do not want to worry about the logging logic right now and and would rather worry about the method logic we are writing. Now this is where we might want to write a partial method. Let try to how a partial method is declared and used.

partial class Foo
{
    public void DoSomething(string someParam)
    {            
        // Log the input
        Log("DoSomething", someParam);
        // Do something important here
        CallSomeMethodSomewhere(param);
    }

    partial void Log(string functionName, string parameters); }

Now this is a perfectly valid code and will show no compilation errors. Since we have not yet defined the body of the function "Log" the C# 3.0 compiler will just ignore our function call. So a partial method lets us define a method signature and call it even when there is no implementation. Also compiler does not emit any IL for the call that we made. So the code shown below is fully ignored.

Log("DoSomething", someParam); 

Now after we are done with implementing out class, we have decided to write the function
and provide a function body. So now our code looks like this ...

partial class Foo
{     public void DoSomething(string someParam)     {                     // Log the input
        Log("DoSomething", someParam);         // Do something important here
        CallSomeMethodSomewhere(param)     }

    private void Log(string functionName, string parameters)     {
        Console.WriteLine(functionName + ":" + parameters);     }
}

So this way we are not changing the original code at all rather providing the implementation of the function later. One use of the function might be to write functions in you class and call them, while someone else may provide the concrete implementation in a function later. This will allow two persons co-ordinate and use virtual placeholders for code. This is a cleaner approach to write to-do functions later rather than providing a method body and throwing not implemented exception later.


Well ... anyway I found this to be a strange feature of C# 3.0. What did you think?

kick it on DotNetKicks.com


Timesaver: Awesome New C# Property Syntax Shortcut

I just could not believe this! No, this cannot be, how did Microsoft know that I wanted this feature!

Well ... today I discovered this new shortcut writing properties in C# 3.0 using Visual Studio (actually I was using the C# Express edition). Lets say we want to have a class called person who has age and name. In C# 3.0 there is no explicit need to declare a variable and write get and set methods for the property.

I could just declare the class like this ...

class Person
{
    public string Name { get; set; }
    public int Age { get; set; }
}

And then use it in code like this

 Person p = new Person();
p.Name = "Me";
p.Age = 31;

This saves so much time. I just love this ...awesome way to define properties. Well I could also use the named parameters style code like this ...

Person p = new Person() { Name = "Me", Age = 31 };

C# has just gone crazy ...

How does this work?

Well if I try to decompile the code using Reflector then we would see that we have this following code for the class Person

internal class Person
{
    // Fields
    [CompilerGenerated]
    private int <Age>k__BackingField;
    [CompilerGenerated]
    private string <Name>k__BackingField;

    // Methods
    public Person();

    // Properties
    public int Age { [CompilerGenerated] get; [CompilerGenerated] set; }
    public string Name { [CompilerGenerated] get; [CompilerGenerated] set; }
}

Looks the like the following code was generated by the compiler ... I am still not clear how the "<PropertyName>k__BackingField" expression is used in sync with the compiler generated attribute, but I am happy that I have such a shortcut.

kick it on DotNetKicks.com

.NET Basics : Explicit vs Implicit Interface Implementation

Nearly all of us know that we can implement an interface in 2 ways, one is explicit and the other one is implicit. In case of the implicit implementation the interface methods stay public and in explicit implementation the methods stay private in C#. Let me show a simple example

interface ICar
{
   void Start();
}

Now this method can be implemented in two ways, the first one is an example of the implicit implementation

class BMW : ICar
{
    public void Start()
    {
        // write code to start the BMW
    }
}

See the method is public, it can also be protected as well. Now the explicit implementation should look like this

class BMW : ICar
{
    void ICar.Start()
    {
        // write code to start the BMW
    }
}

As you can see there is no public or private keyword present for explicit implementation, and if we try to put a access modifier like public or private then we would get a compiler error. However the C# compiler will make this method private.

Well, except for private and public is there any other significance?

No Virtual

Well yes, there is. An explicit method implementation cannot be virtual. So we cannot override the method implementation in a child class. If we had declared the method virtual in the implicit implementation like the code below then we would have been able to override in a subclass.

public virtual void Start();

So you can see the that if we had a specific implementation of BMW that needed a different way to start then we would be able to override the virtual function and call it.

No direct access for Derived class

Also the derived class cannot call the method directly, we cannot call base.Start() in a child class. We would need to cast it to the interface. So a derived class has to write code like this for explicit implementation.

ICar car = this;
car.Start();

No abstract Implementation

I almost forgot another issue, which is ... we cannot use explicit implementation in an abstract form. If we had defined the class BMW an abstract class we could have left the implementation abstract for a derived class to implement like this

abstract void ICar.Start();

But with explicit implementation we cannot delay the implementation in a later derived class.

Hmm ... when is explicit implementation a must?

Well, lets say we had another interface that has the same method with the same signature and then it could be a problem. Let see this example below

interface IBoat
{
    void Start();
}

Now lets assume we have a amphibian car than can run in both land and water but engine for water and land are different so it would require two different methods to start. Since the implicit implementation can have one method the same name in both cases if IBoat and ICar the same method would be called.

class MyAmphibianCar : ICar, IBoat
{
    public void Start()
    {
        // write code to start the BMW
    }
}

This is a problem since we want to have 2 different code to start the engine. So we would have to go to implicit implementation and write code like this

class BMW : ICar, IBoat
{
    void ICar.Start()
    {
        // write code to start the BMW
    }

    void IBoat.Start()
    {
        // write code to start the BMW
    }
}

Now our problem is solved and we can use two different codes for the two implementations.

When is Explicit Implementation a Problem?

If we use explicitly implement a value type then if we need to access the interface methods then we would have to cast the value type to the interface type which would cause boxing and may hinder performance.

kick it on DotNetKicks.com

Back in Dhaka, New Job

I could not make any post for about 2 weeks since I have been too busy with relocation from Kuala Lumpur to Dhaka. I have lived in Dhaka for over 30 years, but one year in KL has made me feel KL as homely as Dhaka, or even more. It is the city bursting with life at each and every moment.

I am no longer working for Pageflakes, a small internet based startup based in San Francisco, CA whose competitors are Netvibes, iGoogle, Live.com etc. I have re-joined Orion Informatics as Principal Software Architect which makes regulatory financial applications for US software firms whose end users are top banks in the US like Washington Mutual, Bank of America, HSBC and also regulatory bodies like US Federal Reserve, FFIEC etc. Since I have worked here for long, I feel like back at home.

One thing I miss most of KL, that is my 1.5 Megabit internet connection! Dear reader, please keep on reading ... I will be posting frequently now.


Multi-core CPU Caching: What is volatile?

I must give thanks to Jeffrey Richter for writing his wonderful book called CLR via C# as it cleared up the confusion about volatile keyword that I had.

How does CPU access variables in memory

We all have heard that CPUs have cache memory and having more cache memory at the CPU (L1,L2,L3) makes processing faster so when we buy a computer we try to buy it with a CPU with large cache memory. Why? The reason behind this is that in terms of CPU speed the RAM on motherboard is very slow. The memory that CPU has called cache memory is extremely faster than the normal memory.

So on the first run the CPU accesses the memory address and stores it in the cache. When the variable is accessed for the second time, it is returned from the cache. So all subsequent reads are done from the cache. The same thing happens for write operations as well. When the variable is changed, it is changed into the cache, also subsequent read/writes are done from cache. However the writes are eventually flushed into memory when cache is cleared or filled with other data. One of the intelligent things that the CPU does is when fetching the value of the variable (a few bytes) into the cache, it also fetches the values around it since the next variables to be used should be close by it.

Volatile1_2

This is fairly fine when you have only one CPU, but most of the PC and Laptops have multiple CPU. Now with the multiple processors this cache access can become problematic. With multi-core CPUs, true threading happens, two machine instructions get executed physically at the same time. Since two processors can have different caches, which are copies of the ram, they can have different values

In x86 and x64 processors (according to Jeffrey’s book) are designed to sync the caches of different processors so we may not see the problem. But the IA64 processor takes the advantage of the fact that each processor has its own cache and does not synchronize rigorously. So different threads executing may have put in different values in the cache.

Please note that one CPU may write to the its own cache, which will eventually be transferred to RAM, the other CPU may have read from the RAM which contained old value since the first CPU has not updated the value yet. See below (click on image for larger view)

Volatile2

So this creates an obvious concurrency problem. That is where the volatile keyword comes in. If you declare a field volatile it is always read from memory and written into memory immediately. However it must be noted that all interlocked operation (again thank to Jeffrey) like lock, Monitor, Mutex, Semaphore etc synchronizes the caches. So this volatile keyword is not needed for those.

But volatile variables are slower. It would have been great if there was a write volatile only variable.

kick it on DotNetKicks.com


CLR Fun Stuff: How is my C# code converted into machine instructions?

As we all know that our .NET code (C#, VB.NET etc) are converted into MSIL instructions which in turn are put into the assemblies. MSIL is a high level language, not as low as machine instructions, so it needs to converted into machine specific binary code or needs to be interpreted somehow. Since interpretation would make the execution significantly slower, the code is converted into machine code on access. This is done via a procedure called Just in time compilation (JIT). Just in time is a management concept that was introduced by Ford Motors into production environment. This is a process where inventory was brought on time just before they were needed, and this saved the warehousing or storing costs.

How does JIT compilation happen?

In programming JIT works like this. Since the executable or the library is made of MSIL (bytecode or any other intermediate form) instruction it needs to be compiled into machine code, but if we convert all of the code into machine format then it will take time. For example if JITwe have an application has 50 functions and we use 3 of them regularly, then if each time when we load the program and compile all the 50 functions then it would be a waste of time and would take a long time to load. What JIT does is to convert the function's MSIL into machine code just before executing the function. See the figure to see how a code is compiled via JIT. Once a code has been transformed into native machine code it stays in memory and next calls to function are pointed to that same memory so that the conversion to machine code is done only once in the lifetime of an executable.

A little deeper look: The secret undocumented CLR function that does it all

When a .NET application loads then it loads the MsCorEE.dll which loads the correct version of the MsCorWks.dll (the version of .NET we are running) which contains all core functions of the .NET runtime. For more detail on loading see the web, one good resource can be this post NET Foundations - .NET execution model. There is this function called _CorExeMain which actually loads the CLR and all the types that are required into memory. There is a memory table for types and there are tables for functions and properties of each types.

Lets say we have a class that looks like this

class TestClass
{
   static void ConsoleAdd ( int value1, int value2)
  {
     Console.WriteLine ( value1 + value2);
  }
}

Now a careful look would tell us that only 2 types are used here TestClass and Console. If we call the TestClass.ConsoleAdd function from the main method this is how the memory looks like before the function is called.

jit2

Before the call both the ConsoleAdd and WriteLine and other functions are pointing to the secret JITC function. This is how each time the JITC functions compiles the code into native code and replaces the function pointer in the function table for the type.

            JIT3

Now lets look at the memory after it has been JITed.

JIT3_001

Last words

So now we know how our C# code is compiled in to native code. It may be hard to believe that at certain times JITed code runs faster than the native compiled code. When a code is JIT compiled it takes the advantage of the exact CPU instructions present the the machine but the native compiled code compiles into more generic class of machine instructions. We will read about advantage and disadvantages of JIT another day.

kick it on DotNetKicks.com