justnbusiness

Moving to a new blog

2007-08-31T06:26:00.000-07:00

I've decided to create my own blog site. I'ts called justnbusiness and that is where I'll be posting from now on. I did it for a couple reasons, one was because I was tired of not being able to upload zip files to blogger. It's great that they let you do images but I really want to be able to do zip files since I regularly want to post code snippets along with the blog. Another reason is because I wanted to have a visible working example of NBusiness in action (even if it is a simple one). I wanted to 'eat my own dog food', if you will.

So if you have subscribed to this RSS feed (why do I hear crickets chirping?) feel free to delete it and head over to http://www.justnbusiness.com and subscribe there instead! This will likely be my last post to this blog.

RC2 Available

2007-08-27T08:03:00.000-07:00

I just uploaded a new release candidate this weekend, cleverly dubbed 'RC2'. This is the first v2 build that will actually install as visual studio plugin and work correctly without requiring the Visual Studio SDK. Be sure to read the README.txt that comes with the build. There were two tricks with fixing the problems with that last RC.

If you don't have the SDK you at least need the ProjectAggregator2.msi, which is a redistributable that comes with the SDK.
I had to get the version information in the registry keys of the installer up to date with the new assemblies. This version has a new PLK that I got off of the Microsoft VSIP website as well. I had been adjusting it in code but forgot the installer required some manual updates.

I plan on upgrading the installer project to a WIX projects as soon as possible. I don't like that the project aggregator redist. isn't part of the automatic install process and it's practically impossible to figure out how to do that with the old school install process. I'm not really looking forward to redoing the installer but I want a double click-next-next-next install process and anything less is unworthy.

There are a few known issues with this release, most of them are listed on the Issues List on codeplex, for example the oh so annoying error message about not being able to get references for your assembly despite the fact that your project will successfully build. I've never seen an error message that lets you build still before. But this is a known bug in the VS SDK, hopefully it'll be fixed on the next release. I guess I'll have to try to get this thing working with VS2008 sometimes soon as well.

Anyway download RC2 and let me know what you think! Someone start up another good thread on the forums!

NBusiness v2.0 RC1

2007-07-31T14:18:00.000-07:00

So it's here at last! This is the not quite officially released version of NBusiness 2.0. Here is the link:

http://www.codeplex.com/NBusiness/Release/ProjectReleases.aspx?ReleaseId=1062

For those of you who have tried out my Alpha/Beta versions of 2.0 in the past I'll say that this is a much easier install to get working. This is the real-deal, no experimental mode BS to have to deal with. Full integration baby!

I should say that you should expect some bugs for now but it should be "mostly" working. Meaning, simply firing up a new project should just build right out of the box.

Most core features should be working just as you'd expect but I haven't tested out the VB side of things thoroughly enough yet so don't expect much from there. If you test it out and find some bugs please enter an issue in the issue list! I'd really appreciate it.

After doing some thinking about the time spent getting to this point I realized that I haven’t been recording hours at all and it doesn’t seem like TFS keeps track of checkout times so there probably is no way to know for sure but I’d estimate that it has taken me an average of 10h/week (not including the time spent working on the XNA game I made recently too!) for 10 months which is about:

10 * 4 * 10 = 400 hrs (or 16.7 days)

If I was billing at $90/h that would be $32,400!

Of which I’ve made approximately $0 total. I’m hoping there will be some intangible benefits down the road but the main reasons why I did it were A.) to satisfy my thoughts on how to make a code generator and B.) I wanted to make a website and resented writing the data access layer by hand. So basically it’s taken me almost a year to write some code that would have taken me just a couple days if I wasn’t so damn crazy. Talk about taking the long route to solve a problem!

After squaring up the VB side of things I need to make sure the Getting Started doc is better and I have to update the wiki pages so they're in line with the current version. This is pretty exciting! Be sure to tell your friends!

Ghetto Alarm

2007-07-27T17:57:00.000-07:00

Last night I found myself without a cellphone or an alarm clock, I needed to get up by 8am so I had to come up with something. Of course, being the nerd that I am, I decided to write up a quick alarm clock application before I fell asleep. I just plugged in my laptop and changed my power settings so that it would never go into sleep mode and executed the following program:


using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Runtime.InteropServices;

namespace GhettoAlarm
{
    class Program
    {
        [DllImport("Kernel32.dll")]
        public static extern bool Beep(UInt32 frequency, UInt32 duration);

        static void Main(string[] args)
        {
            DateTime alarm = new DateTime(2007, 7, 27, 8, 0, 0);
            while (true)
            {
                Console.Write("\rTime: " + DateTime.Now.ToShortTimeString());
                Thread.Sleep(500);
                if (DateTime.Now > alarm)
                {
                    Beep(500, 250);
                }
            }
        }
    }
}

Needless to say it worked perfect! I'm not sure what would have happened if the laptop power mode was set to fall asleep but I think it probably would still work.

I think the interesting thing here though is how easy it is to just write up a little something to do what you want in C#. I know there is some talk going around about the new dynamic languages and how they are great for rapid prototyping but I just want to say with C# you can change which line of code is executing on the fly, change it and recompile on the fly and even change values in memory on the fly. I'm pretty happy with the rapid prototyping capabilities of C# and I'm a little skeptical of the value of dynamic languages in general... One of these days I'll have to try python again before being too certain about this though.

CodeSnippetExpression is evil

2007-07-09T12:32:00.000-07:00

By the way, to anyone thinking about doing some CodeDom development be sure to avoid the CodeSnippetExpression class like the plague. It turns out there is (at least) two legitimate usages of the class that I have encountered but otherwise it is evil.

I can say this because I just spent a couple weeks re-doing all of my templates so that the CodeSnippetExpression class was completely removed from my templates. This was only the beginning of the headache of making my CodeDom code language agnostic but it was the most painful. For the record I'll add my two legitimate usages of snippets and some examples of how I got burned by just saying to myself "Meh, I'll only be generating C# anyway...".

Legitimate use #1 - Calling the default base class constructor

//Example
CodeConstructor c = new CodeConstructor();
c.Name = "Example";
c.Attributes = MemberAttributes.Public;
c.BaseConstructorArgs.Add(new CodeSnippetExpression(""));

//Output
Example() : base() { }

For some reason you have to add a parameter to the BaseConstructorArgs collection to get this functionality and if you want to call the parameterless constructor you have to add an expression that generates nothing. I suppose you *might* be able to use the CodePrimitiveExpression(null) but I haven't tried that. I can confirm that the empty CodeSnippetExpression("") does work however.

Legitimate use #2 - Fake while loops

//Example
CodeIterationStatement i = new CodeIterationStatement(
new CodeSnippetStatement(""),
new CodePrimitiveExpression(true),
new CodeSnippetStatement(""));

//Output
for(;true;) { }

Now I realize that this looks stupid but I assure you, it's legitimate. It turns out there is no "while" loop in the CodeDom all you get is the CodeIterationStatement which in turn generates a for loop. However if you think about it what is a for loop? Essentially you're saying "for some initial value, while some condition is true, do this". A for loop with only the condition evaluation is essentially a while loop and the only way to do that with the CodeDom is to print out empty snippets.

So it's ironic that the only legitimate usages of the CodeSnippetExpression objects are simply empty strings. This makes me think that they're utterly evil and should instead be replaced with a CodeEmptyExpression() object or something similar. I welcome any comments about how else they could be properly used.

Here are a few simple examples about how I decided to use them then found it to be very painful to undo them:

Bad idea: "using"
Yes I know the "using" feature in C# is lovely and wonderful and all of that but in reality it is simply syntactic sugar. It is not part of the underlying .net framework.

Proper Alternative:
IDisposable d = ...;
try
{
//do something to d.
}
finally
{
d.Dispose();
}

This is essentially what the "using" statement is doing under the hood. Unfortunately you have to implement this using the CodeDom this robustly. There are no language agnostic shortcuts!

Bad Idea: "anonymous delegates"
Anonymous delegates, oh anonymous delegates, how I love thee... unfortunately you are simply an illusion! Once again more C# syntactic sugar. What's going on under the hood is that a private member is getting generated and a delegate pointing to that method is being generated for you. You're going to have to do it all yourself! Additionally, simply passing method names as parameters for delegates isn't good enough either, more damn syntactic sugar! You're going to have to do it all the hard way.

Proper Alternative:
void Test()
{
EventHandler e = new EventHandler(Test);
e.Invoke(this, EventArgs.Empty);
}
void MyDelegateMethod(object sender, EventArgs e)
{
//do something.
}

This goes for asynchronous methods as well. In stead of simply passing in Method names for parameters you'll need to use the CodeObjectCreate class to create an instance of your delegate (EventHandler or whatever) and pass in the Method name to that. Make sure you use all the proper CodeDom classes to get this language agnostic.

Those were really the biggest gotchas that I found out. You should start out assuming you're going to be generating for multiple languages and do it right the first time. Be sure to find the right class for the right job too!

Nearing v2.0 Release

2007-06-19T08:10:00.000-07:00

I only have 2.5 work items left before the release of v2.0. And I may defer one completely. I'm having a bit of a hangup regarding whether or not to implement a feature where you can generate E# based on a database schema. It's not the technical challenge of it, though it would be a fair amount of work, I have done it before and know exactly what it would take to do it again. It's the whole idea of it that I'm debating with. Yesterday at work I was giving a Lunch-and-Learn presentation about code generation and one slide of my power point presentation I have labeled "Source of Truth". This slide prompts me to talk about how I view the meta data as a source of truth and, really, this is the main difference between NBusiness and all other code generators. I'll explain what I mean a little better.

To me with code generation it is possible to end up with a somewhat schizophrenic code generator when it comes to keeping track of it's meta data. This is what I mean by the "Source of Truth", where does the meta-data really reside? For most other code generators they start with extracting the schema from a database as the source of truth and this is the primary problem. However, I question the value of pulling in the schema from the database as a source of truth.

The main problem is really that, since you're pulling it in from a database you are trying to map Objects based on a relational model. This is a well known process known as ORM and as we all know it is a very complicated and tricky problem. However, no matter how well you do ORM, it simply isn't possible to express everything you need in an object in your relational database (at least in a way that translates into an interpretable schema) and inevitably there is a need to customize or add to your meta data. And here is where the problems begin, you have two sources of truth suddenly. How do you handle it so they merge together smoothly, how do you keep customizations while trying to recognize changes in the database, how do you effectively retrieve a subset of the database schema, how do you give things a more friendly name while trying to keep them mapped to the original column and more questions? I mean most of these problems have solutions but they are complicated and difficult and imperfect at best. You might include not allowing customization at all and just forcing the user to deal, you could use some mechanism to merge customizations in with extracted schema meta data.

SO with all that said I should explain how NBusiness differs. NBusiness is a reversal of where the source of truth resides because your E# files are actually your source of truth. They are the 1 and only source of meta data, and to complete the reversal it turns out it is FAR easier to generate your database schema based on objects than the reverse. You could say ROM is much easier than ORM. So for starting up a new project, hands down I'd say the model of authoring your meta-data (entities) without being dependent on a database for extracting schema is far superior. What it comes down to is tooling at that point, which is where I believe NBusiness is unique and valuable. That is, rather than a graphical Visio style development tool you have E#, which is a very easy and intuitive (or strives to be!) language to describe meta data. I think this is why most people have gone the ORM route, because most databases already have a rich and familiar tool for creating tables and columns and relationships, why reinvent the wheel? Well it's my main supposition that E# turns out to be much easier and productive for describing authoring meta-data and you also end up becoming independent from the database as a source of truth which is very valuable.

While I believe all this is true, there is still the situation where you have an existing database. Also, it may be fairly large and manually creating all these entities may be more work than you really want to have to do. That alone could be a critical enough point where using a different tool than NBusiness is a superior choice. So it still might make sense to have the capability to generate some entities based on a schema to really get you started at least. If I did it I'd do it with the focus on keeping your entities as your primary source of truth still. I sort of want to do it but I don't want to degenerate into just another ORM tool. Though, I suppose, if I provide that feature than it will really put my theory to the test. If you have the ability to generate your entities from a database AND the ability to define your entities as the single source of truth then which ever method is truly superior will be the way people use it. If authoring E# manually is not actually better than that tool will be helpful, if it is then it will be helpful sometimes and not other times.

I think I'm sort of talking myself into this even though I set out to talk myself out of it! Dang it.

Maybe I should speak to how I envision NBusiness to handle the situation where you have a set of entities that do not map 1:1 to a database. I imagine this problem will crop up often and I haven't fully tested out the system yet but this is how I imagine it will go. Essentially, when you generate your code you may optionally generate SQL, this sql you get for free and if you're working on an existing project it's really helpful. However if you're not and you have an existing database you may end up in a situation where you don't get to use any of that sql at all. So with NBusiness all of the code it generates (with the default templates and the default framework) you end up with code that uses Stored Procedures as the bridge between the objects and the relational data. All NBusiness code ends up calling a stored procedure such as "PurchaseCollectionFetchByPerson", which when authored would give you a PersonId parameter and expect a set of data related to a collection of Purchase objects. That stored procedure may query tables that look nothing like the purchase entity you have declared so it is the responsibility of the stored procedure to be that bridge. You would have to manually create this in this circumstance.

This get's us 90% of the way there very effectively I think, optimally even. Though there is still the situation where you have a terrible database schema that is not what you want to map to your entities 1:1 AND you don't have the ability to create stored procedures. Maybe your database doesn't have the capability (like older versions of MySql) or perhaps it's just a policy of your company for some reason. For this, unfortunately I'm not sure what to do exactly. I can think of two solutions, one would be to create a custom database provider and translate those stored proc names into some sort of sql but this stinks of a hack. The other solution might be some sort of [Attribute] system in your entity definitions that tell the code generator how to formulate non-standard queries. Actually, I rather like that idea... It probably wouldn't be that hard at all. Well there's something for me to noodle on for a while. v3.0 maybe, LOL.

Anyway, back to enjoying my summer afternoon (or just before noon)!

The 5 Laws of Code Generation

2007-06-07T08:14:00.001-07:00

A colleague of mine from Magenic has written an blog about code generation which summarizes a conversation we've been having:

http://blog.magenic.com/blogs/jons/archive/2007/06/06/Thinking-about-Code-Generation.aspx

I really like this post because it nicely lists all of the aspects of code generation that I have been thinking about but haven't been able to put into words. What's nice is that I've tried to address each of these problems with NBusiness in one way or another. Some of these solutions are somewhat unique, especially in terms of the Template system. The following are descriptions of how NBusines resolves the issues outlined by Jon in his post.

CSLA places most of the "heavy lifting" in a set of common base classes

NBusiness does not use CSLA, instead it comes with a light weight, medium-trust, business object framework. It mirrors most of the core features of CSLA as a business object framework but does so in a very different way (due to the need for medium trust). However there is no reason why NBusiness couldn't be used in conjunction with CSLA (or any other framework) since it is template driven but I wanted the official distribution to be completely self contained for now.

I have some fairly specific "technical requirements" that I think that code generation has to to provide:

To me this list is like the "10 commandments" of code generation. Only there are 5, so let's go with the "5 laws of code generation". I'll summarize his list and try to explain how NBusiness conforms to these laws.

1. Code generation is controlled through modifiable templates.

In NBusiness everything is generated by a template. Rather than using any of the existing template schemes (as far as I know) NBusiness takes a unique, more object oriented, approach. The templates themselves are actual classes that inherit from a Template base class. They contain a single method with the signature "ResourceFile[] Generate(Entity[] entities)". The resource files contain a byte[] and path for the resulting files (you could concievably generate images or other binary resources based on entities this way). Of course what goes on in the Generate method is completely up to the template developer. Furthermore, this scheme allows for inheritance in templates. So, for example, if you decided to use one of the templates that came with NBusiness but you just don't quite like one line of the output, you can create your own template and override an appropriate virtual method and customize the output yourself without destroying the original template. This more object oriented approach brings forth some interesting possiblities.

There is also a notion of template "dependencies", which allows you to selectively generate code if an Entity contains a template that fulfills a dependency. For example if you have a template for generating an EntityBaseCollection object, you'll naturally need an EntityBase object to be generated as well. Therefore the EntityBaseCollectionTemplate is dependent on the EntityBaseTemplate. This problem arises quite a bit and this solution is very helpful for solving these issues. Without it you end up getting to a point where you might as well just make one giant template and apply it to all of your entities rather than selectively implementing only what a particular entity needs.

2. Code generation is done during the build process.

This law is one of NBusiness' best features. Not only is done during the build process the NBusiness engine is actually a compiler. In fact the artifact that NBusiness produces is a class library rather than simply a set of classes to be loaded into a seperate class library project. In v2.0 this is automated fully by means of Visual Studio Integration. You litterally have an E# project that can be referenced by your other projects as intuitively as you would any normal class library.

3. Code generation is done from an intermediate format.

This law is very important to me. NBusiness goes one step further than what Jon is suggesting as the optimal solution however. Jon suggests using XML that is (optionally) generated from a database schema, which I couldn't agree more as the right solution to the problem. However instead of XML NBusiness features an Entity Definition Language known as E#. It is essentially the same thing: a set of meta data that describes your entities. However it is my belief that XML really isn't as developer friendly as what is really needed. Rather than creating all sorts of grahpical development tools that allows a developer to manipulate this XML what is really optimal is to have a model that is the same as what most developers are currently most familiar and productive with, which is code. E# (in v2.0) has visual studio integration which comes complete with syntax hilighting and intellisense and allows a developer to build their entity meta data in a style similar to what they are already familiar with. The only barrier is that you have to learn "yet another language" to do it. E# strives to be extremely simple by having syntax similar to C# and it is also fairly VB-ish in how the keywords flow. Intellisense helps quite a bit as well of course.

4. The intermediate format is under source code control with versioning.

This law is helpful for restricting the definition of what the intermediate format is. I would suggest an ammendment to this law that states the intermediate format must also be such that it is compatible with "merging" features of source code control. This will promote the ideal of having this data as plain text rather than a custom binary format. Of course, since E# is defined in source code files it meets this law fully.

5. The code generation templates are also under source code control.

Again I would ammend this with the mergability rules from #4. In the case of NBusiness the base templates are source code files currently under source control at CodePlex while custom templates would simply be class files somewhere in your project, which is absolutely compatible with source code control.

The next section of Jon's post addresses more of the features required of a business object framework as they relate to code generators. This is an important distinction because, while NBusiness has such a framework, it could certainly be used to target any 3rd party business object framework which may or may not conform to these rules. This is mostly a concern of the templates but there are definitely accommodations for a particular style that need to be taken in the framework itself. NBusiness' default business object framework takes the second style, where partial classes are used to accomodate custom code.

To me the main benefit of this is the cleanliness of the resulting object model. While I'd concede that the base/child style is functionally fine, I would argue that having fewer classes is far superior. The tradeoff, is that you end up having to expose many events in order to provide extensibility points for the other parts but this is ok to me. In fact in NBusiness there is a feature called "actions" that allow you to define the behavior of an entity as a method. These actions can optionally be attached to these extensibility events and even be automatically executed asynchronously. This action method is implemented by the developer in a partial class or even as a static method on another class entirely. So these events would be there anyway.

An example of this might be a "TransferMoney" action on a BankAccount entity. Or a Log method attached to the "persisting" event, etc. You can envision how useful it is to define your entites behaviours in the entity definition itself. The architect can define everything and the code monkey can implement it.

So with Jon's post in mind I propose the 5 laws of code generation:

Code generation is controlled through modifiable templates.
Code generation is done during the build process.
Code generation is done from an intermediate format.
The intermediate format must be versionable and mergeable.
The code generation templates must be versionable and mergeable.

Bugsville

2007-05-12T00:38:00.000-07:00

DOH!

I have been stumped on my VSIP work for the last week or so. Progress has been very slow fixing the code I stole from the IronPython project to get my "simple" language project working. It's very hard to debug and the code get's very dense very quickly. I learned something new today too, which is pretty surprising: don't rely on quotes when searching google.

I had been searching for a quoted block of text for my error message ["Could not get dependencies for project reference"] to try to locate some information about this mystery bug. This is the literal text of the error message. You'll notice there are about 5 results, none of which are very helpful. After a week of digging through code and yielding nothing I scrapped together a different query [Could not get project reference web], I got rid of 'dependencies' and 'for' and added the word web and low and behold we get this link in the first page. If you check out the "actual results" section on the bug report you'll see the quoted string "Could not get dependencies for project reference"! Which was my EXACT quoted query to google. Yet it did not yield results. Just goes to show you, a few select words is really the sweet spot for google. Their advanced query options aren't very good.

This might be an ok time to continue griping about Google. Don't get me wrong I really like Google overall and their website is still my home page but I do have a few gripes.

1.) When I was in college I had an undergraduate research project (don't laugh, I don't have permissions to edit that page anymore :-P ) that used the Google API. It was fun and a really great way to learn about web services and .NET and at one point in time I was featured on http://code.google.com/ as the featured project since their API was relatively new (this was just before Google maps API) and the project attempted to use Mono as well. Anyrate, so because I was featured on their page they offered to send me a Google T-Shirt as a gift. I thought that was pretty cool but no tshirt ever came. After harassing Chris DiBona at Google for almost a year he finally sent me the shirt as well as a spiffy lava lamp (that was the year of lava lamps since I inherited one from a friend, got one from Google and got one from my dad for Christmas... go figure). So they made some amends, which is good, but it's worth mentioning that waiting for a year for a Google T-Shirt is torture.

2.) The Google API is a scam. It's essentially a marketing ploy. As I mentioned in number 1 I worked on the Google API for an undergraduate research project but beyond that, if you go to the Google API usenet group you'll notice that I'm the #1 all time poster by far. Actually Manfred seems to be catching up. But anyway, the first half of the posts were me geniunely caring but the second half were me pretty much answering the same 3 questions over and over again:

"I can't download the Google API zip file, can someone email it to me?"
"Why are the search results different from the API than from the web site?"
"How do I use the webservice in language X?"

Ok, #2 is the real problem. Why are they different? Well because they don't use the same databases for the API as they do for production. Which is incredibly lame, what is the point of only being able to query a subset of the data? There was one guy who was running the exact same query over and over every day and tracking the changes, the API never changed and the website was changing dramatically every day. The were essentially just using a set of junk data they ripped from their main databases but was never updated. Anyrate, there is a (relatively) new development you can read in this thread saying that the Google SOAP API has officially been shutdown in place of some AJAX search API. I'm not sure what the difference is... anyway it was really dissapointing to realize how lame the search API was after putting so much effort into it.

3.) What the hell is up with their desktop application suite? Is it just me or is it the biggest pile of crap? It totally goes against their sacred standards of simplicity. It is one of the most monstrous conglomerates of useless applications. In fact this has been a Google theme of late. Their iGoogle homepage site has become more and more bloated by widgets and tabs and themes and what not... UGH, please make it stop! The only reason I use that iGoogle site is so I can get a quick link to my gmail inbox and a quick wikipedia search box. Please Google, return to blissful simplicity! Let Microsoft bumble around with their bloated search home page, if you don't I'll be forced to switch to live.com, which doesn't quite have the quality search results but it loads up as a homepage much faster. I don't want a million javascript widgets that drag and expand and move and crap all over my homepage! I want it to load quickly and provide me an opportunity to get where I want in a few clicks.

I feel like an old man complaining like this :) Anyway, I do have some great news to report for NBusiness (other than that damn SDK error). I have rudimentary intellisense and syntax hilighting working, I have the ability to add a reference to NBusiness through the ".net" tab of add a reference dialog and I have nested .cs files based on the names of the files (notice the expanded Company.es file in the solution explorer. I haven't quite figured out how to hook up the icons to the filetypes however. It seems to be pretty tricky, lot's of registry keys that are very strange. There are plenty of things left to resolve but hopefully I'll get a usable beta up on codeplex this weekend. Here are some screens:

Visual Studio Plugin Breakthrough

2007-04-12T19:41:00.000-07:00

Well I've been working on a visual studio plugin for NBusiness here since v1 was released. I've encountered a few small bugs and have a host of changes I hope to make to the compiler for v2 but I've mainly just been focused on getting this plugin "working". There is a seemingly infinte list of things I need to fix still but I had a major breakthrough today in my progress!

I'll try to lay out what I'm hoping the plugin will do and give a few nice little screen shots of my current development environment then I'll spend a bit of time ranting about the process of creating VS plugins and I'll even try to drop a few gems of advice for anyone trying to do something similar.

First I'll give the most basic requirements I had for this plugin:

I want to be able to click File->Project->Visual E#->Entity Library
I want to be able to add and edit es/cs files directly to my entity library. (Also add embedded files).
I want to be able to add references to my entity library project from other projects.
When I click build the end result yields a .dll that other projects get by reference.
I want intellisense and syntax hilighting.

Here is a snapshot of me adding an Entity Library via the File->Add New->Project menu!

Here is a snapshot of my dev environment with everything else hooked up (except intellisense and syntax hilighting):

So to get this far I basically just opened up the IronPython example that comes with the visual studio SDK and looked over every square inch of code. There were a couple minor areas of confusion that ended up turning into extreme time delays that I'll warn you about here.

For one thing beware the confusion of naming your language! I had "Visual E#" and "ESharp" and "NBusiness" here and there and nothing seemed to work right. In certain areas it expects the names to match up and if they don't then good luck figuring out what is wrong... Basically I would reccomend simply picking one name (like "ESharp") and using that everywhere. Then after things are working right start tentatively renaming things so they look pretty. That will save you enormous amounts of time.

Another extreme area of confusion was how the heck you get the .zip templates into the right location! Well I was a little lucky because I know a thing or two about MSBuild but in case you weren't aware, it turns out that it is possible to put custom MSBuild tasks manually into your project file (via notepad for example). This is what they did in IronPython and unless you're really paying attention it almost seems like magic how things end up where they do. This is also how they get the .targets file in the right location during build time.

Also, I know it seems ridiculous how Microsoft developed the bulk of their (useful) managed API for VS as plain C# files rather than compiling it for us into an Assembly but trust me, it's worth just adding the damn files to your project and building them. The sheer amount of code you would have to write is staggering, so much seemingly useless cookie cutter code... I have no idea why it isn't just an assembly you reference. But also beware that some of those project files litterally have python specific strings in them! Watch out for embedded resources being loaded by invalid paths due to namespace differences and do a vigorous "Find All" for "py". Get it all out of there!

I'm trying to look over the carnage that is my Visual Studio plugin to see if I can relay anything else useful but all I can think of to say is that if you're planning on creating a custom language for Visual Studio be sure to either have someone who already knows how to make them help you or plan about a month of hard core hacking to get the basics working. This has probably been the hardest (managed, I've seen nastier C++) code I've ever had to work with... I'm going to lose sleep just thinking about what I'll need to do to get intellisense and syntax hilighting to work!

Oh yeah I'll probably be cooking up a beta installer for this pretty soon if anyone wants to try it out. Stay tuned for new blogs or just IM me.

All bitching aside I would just like to say that having this power integrated directly into VS is pretty exciting. When I first got it to build my entity I was litterally shaking slightly afterwards. This really opens up a whole new world for this project and will hopefully bring some legitimacy to the idea of it really being viable. I definitely look forward getting this plugin out the door!

Version 1.0 Released!

2007-03-20T19:49:00.000-07:00

Finally!

Head over to http://www.codeplex.com/nbusiness to check it out.

After touching up little issues here and there for the last couple of weeks (and getting side tracked by XNA) I have finally managed to roll out a v1.0 release of NBusiness! This version is useable but almost certainly not perfect. I can already see a few areas in need of improvement such as the cacheable collection and even optimizing the compiler itself. No doubt the templates will be endlessly configurable, not to mention creating entirely new templates.

Despite how happy I am to have this project at this stage ,so I can finally start some of the more interesting features I've been putting off, I am mostly glad because I actually want to USE NBusiness for my own projects (I have a kegbot project in process). In fact this is why I started this project to begin with, I had been kicking around the idea in my head for a while because of some of the things I had been doing at work (working on a business object code generator), reading a compiler book at some point and then realizing that I wasn't going to be able to use CSLA (www.lhotka.net) on my go daddy website because of medium trust issues. I then decided to just do it, I wrote almost all of the business objects for that site by hand and just got tired of it and started making the compiler. Now I could crank them all out (sql included) in less than a day. And look it only took me about 6 months longer than if I had just done them all by hand! But seriously, I feel pretty good about NBusiness, and I hope others do too.

I've made a branch for v2 already (in anticipation of bug fixes for v1) and have begun work on a Visual Studio plugin. If anyone has had any experience working with language services and custom project types for visual studio please let me know! This is unbelieveably complicated stuff and I am wading through it all by myself at this point. It would be really nice to see Microsoft produce a purely .NET visual studio SDK. The plethora of COM interfaces is absolutely mind boggling... The Iron Python example is enormously helpful but it is also enormously complex. I would like to see a step by step example of how to get a simple Iron Python like project created.

Also, I would like to give a special thanks to Kyle Leneau, who did all of the artwork and helped make the example site look a lot better. Also his interst helped fuel my interest, which is very helpful when you're working on something as long as I have been. Also, thanks to Rocky Lhotka for helping bend my mind which drove me into creating this project to begin with and thanks to Ben Reichelt for taking moments here and there to discuss some of the more arcane aspects of business objects and databases. But no thanks to Beau Crawford for holding onto the managed data access layer he promised me! :-P Also, thanks to everyone else who tried it out at various stages and gave me feedback.

Custom Actions

2007-01-24T16:34:00.000-08:00

So I just finished adding in the final main feature of the E# language. I've been kicking around a few other ideas but have discarded them (at least for now) and I'm pretty sure that this is it. Actually, I still have to add the ability to declare an "expression" but that should be pretty simple compared to everything else.

So the new feature I'm talking about of course is the "action" declaration feature. This brings me one step closer to the philosophy that Rocky Lhotka has been talking about, which is that business objects should be defined by behavior and not data. In this case, as you're designing your entities you can specify the behavior in a lot of ways and one of them is by declaring an action. Here is an example:

family Business
{
    entity Example
    {
        action DoWork when changed;
    }
}

In the above example we declare the action "DoWork" and specify it to be executed when the entity has been changed. There are about 20 reserved event keywords (such as deleting, deleted, fetching, fetched, persisting, validating, accessed, etc.) that you can attach your actions to. Also you can choose to not attach them to anything at all by leaving off the when clause entirely. So here are some other examples of declaring actions:

action Log on Business.Custom.Utilities when changed;
action ProcessMessage when persisted async;

The "on ..." syntax specifies the object in which the expected method of this action is to be found. This requires the actions method to be static, whereas not specifying an "on" clause requires the method to be part of "this", which means it must be defined in a partial class. Also the "async" specifier will execute that event asynchronously rather than the standard synchronous execution. The "async" keyword only applies to actions with a when clause.

So you might ask yourself why you would want to specify actions? Well it's pretty nice really, the whole point is that when you are wearing the architect hat you want to think about your problem not so much as just what types of data you need to pass around but as what types of behavior you want fulfilled. In this case as you are architecting your entities you would be able to specify which actions are exepcted of it, beyond the behaviors implied by persistence, validation etc. The methods that these actions attach to are not generated by the E# compiler but instead they must be manually created. If they do not exist the compiler will complain and tell you that a method is missing. So as the architect you can specify the actions that the developers then have to go and implement them.

Next you might ask, well what is going on under the hood? It turns out to be pretty simple really. The hardest part is just getting all the dang events coded and executing at the right times in the EntityBase class but what ends up happening from the templates perspective is that a method is implemented called Load_Actions and in there event handlers are added to the various events pointing to the specified methods which gets called at construction time. So if the method is not implemented the C# compiler will complain about it. The only trick is for actions that have no when clause specified. For this case I have added a protected event called Never, which the methods get attached to. The Never event never gets called (as you can imagine) but it will get the C# compiler to complain about missing methods! So that's my little hack of the day.

Template Dependencies

2007-01-09T22:29:00.000-08:00

So I decided to implement a template dependency system. The idea is that templates are sometimes dependent on other templates so rather than just having the whole thing blowup whenever the dependent template does not implement something I decided to create interfaces that would act as "contracts" between templates. So for example the EntityInfoTemplate implements the IInfoObject contract interface, which has the signature:


namespace NBusiness.Templates.Contracts
{
    public interface IInfoObject
    {
        CodeMemberMethod GetInfoObjectMethod(Entity entity);

        CodeMemberMethod ToInfoArrayMethod(Entity entity);
    }
}

And since the EntityBaseTemplate is dependent on the presence of the EntityInfoTemplate in order to implement the GetInfoObject() method it can now use the Dependency.GetContract( ... ) method to determine if the EntityInfo template was assigned to the given entity and if that template implements the IEntityInfo method as required. So in templates where a template is dependent on an entity to have certain other methods/fields/properties/classes you can use the Dependency object. For example:


#region Create GetInfoObject
protected virtual void CreateGetInfoObject(Entity entity, CodeTypeDeclaration Declaration)
{
     IInfoObject getInfoObject = (IInfoObject)Dependency.GetContract("EntityInfo", typeof(IInfoObject), entity);
     if(getInfoObject != null)
     {
          CodeMemberMethod m = getInfoObject.GetInfoObjectMethod(entity);
          Declaration.Members.Add(m);
     }
     else
     {
          OnWarning(new TemplateWarning(
               this,
               this.GetType().GetMethod("CreateGetInfoObject", BindingFlags.NonPublic  BindingFlags.Instance),
               Resources.IGetInfoObjectWarning));
     }
}
#endregion

Changes to the templating system

2007-01-05T18:43:00.000-08:00

So I've been struggling to find the best way to work out the template system and even though I'm not 100% certain that I found the best way to do it I think it is pretty good right now and I actually feel satisfied with it. Though I would like to come up with some ways to have a "control" like system for templates (reusable parts of templates) I'm just not sure how to do this yet so I'm still happy with 1 big template. So I made some fundamental syntax changes with the language and I sorta like it.

Previously you would have to specify which templates you wanted to use at compile time, then it would take all of your Entities and put them into each template. I toyed with having different template types such as a Family template and an Entity template where it would put each family or each entity into the template but I just never liked that much. Not only was it not elegant enough but it also ended up being a bit confusing and didn't solve any issues of strong coupling between templates or limitations with wanting a template to apply to only specific entities. So here is an example of the new system:


family Example
{
     template Custom as Example.Templates.Custome in Example.Templates;

     entity Test as Custom
     { }
}

In this example the template is explictly declared in the family,
this is one of the new changes. Additionally, the entity is assigned templates using the "as" clause. Currently it must only use templates found in the same family, though it might make sense to allow you to specify the fullname of the template to get to templates in other families.

So during compile time the template will be created by the Activator class and the list of all entities defined for that template are passed into it. This gives the developer the ability to specify which entities specifically get passed to the template rather than an all or none sort of deal. Additionally it gives the template developer the ability to see what templates an entity is assigned to as well as entities in relationships. So if, for example, a particular entity does not use a "list" template then your template should not create a list property for a type that will not be generated.

For example, suppose you want to create a method on your EntityBase classes GetInfoObject(), which will return a simple Info object representation of your class. Now in the old system you've just coupled your template together with an EntityInfoTemplate, which makes me ask the question why have seperate templates at all? Why not just have one template that spins through the families and generates everything just the way you want it? Well that just isn't as flexible and extensible as I would like. So in this new system you are optionally coupling yourself to the info object template because you can actually look to see if your entity calls for an InfoObject template before you create the method which would return a type that will not be generated.

Like I said, I'm not 100% certain this is the perfect way to do this but I think it brings me a little bit closer to the nebulous "control" design pattern for the template system that I cannot quite envision yet.

Dynamic sorting and paging

2007-01-05T08:21:00.000-08:00

I ran into an interesting problem recently, trying to allow my EntityDataSource control to implement dynamic paging and sorting. There are some tricky ways to do either one fairly easily but to do both at the same time? Well let's just say it's not trivial. SO there might be a better way and I'd be happy to find out from others how to do this more simply but here is a snippet of how I got it to work (for sql2005):

ALTER PROCEDURE [dbo].[CompanyListFetchByRegion]
@RegionId int,
@StartIndex int = 0,
@MaxRows int = -1,
@SortField varchar(35) = null,
@SortOrder int = 0,
@TotalRows int = 0 output
AS
BEGIN

SET NOCOUNT ON;
DECLARE @Temp TABLE
(
TempId int IDENTITY(1,1) PRIMARY KEY,
[CompanyId] int,
[Name] varchar(50),
AddressId int,
Url varchar(128),
Phone varchar(16)
)

IF @SortOrder = 2
BEGIN
INSERT INTO @Temp ([CompanyId],[Name],AddressId,Url,Phone)
SELECT C.*
FROM [Company] AS C INNER JOIN [Address] AS A
ON C.AddressId=A.AddressId
WHERE A.[RegionId]=@RegionId
ORDER BY
CASE @SortField
WHEN 'Name' THEN cast (C.[Name] as sql_variant)
WHEN 'AddressId' THEN C.AddressId
WHEN 'Url' THEN cast (C.Url as sql_variant)
WHEN 'Phone' THEN C.Phone
ELSE C.[CompanyId]
END
DESC -- descending
END
ELSE -- Sort order == 1 or 0
BEGIN
INSERT INTO @Temp ([CompanyId],[Name],AddressId,Url,Phone)
SELECT C.CompanyId, C.[Name], C.AddressId, C.Url, C.Phone
FROM [Company] AS C INNER JOIN [Address] AS A
ON C.AddressId=A.AddressId
WHERE A.[RegionId]=@RegionId
ORDER BY
CASE @SortField
WHEN 'Name' THEN cast ([Name] as sql_variant)
WHEN 'AddressId' THEN C.AddressId
WHEN 'Url' THEN cast (Url as sql_variant)
WHEN 'Phone' THEN Phone
ELSE [CompanyId]
END
ASC -- ascending
END
-- Get the total row count, used for paging.
SET @TotalRows = (SELECT COUNT(TempId) FROM @Temp);
-- If paging is specified
IF (@MaxRows > -1)
BEGIN
SELECT * FROM @Temp
WHERE TempId >= @StartIndex
AND TempId < @StartIndex + @MaxRows + 1
END
-- Else return it all!
ELSE SELECT * FROM @Temp;
END

Good luck!

2006-12-23T07:39:00.000-08:00

So I'm coming down to a real beta1 release. I've been polishing things up and trying to get some bug fixes for everything I can find and stream line the example web site for demo purposes. I even cooked up some graphics and a logo! Check this out:

So I'm planning on using this as the new logo for now, I like the pleasantly corny one liner "Now we're NBusiness". I am no artist by any stretch but I'm reasonably happy with it so far I'm thinking that the N should maybe be twisted just a little so that it appears to be printed on the briefcase rather than floating on it... but i don't have the tools or the skillz to do that. If anyone thought that they might want to try it out let me know and I can email you the layers (you can also get them out of the source code at Code Plex).

I'll probably release the beta on tuesday, after christmas to help maximize exposure on codeplex... but then again maybe no one else would be releasing for the next couple days and it would remain on the newly released list longer. But if no one is even looking... hmm.

So I had a realy interesting problem spring up the other day that turned out to be pretty tricky to solve. I haven't had a fresh computer science problem in a while! It turns out that when you have even a moderately complex object graph it is pretty easy to end up in a never ending loop during persistence. Meaning if you call Save() on a particular entity it will then save all of its parents, itself then all of its children. Which is what you want really but then the children try to save their parents and parents their children etc. So you can end up going round and round forever if your ancestors share a common parent with you. I'll show a diagram of the DB scheme from the Example website to illustrate what I'm talking about.

So you can see there if you're saving a Company entity it circles back around to it's parent Address and you get stuck in an infinite loop.

So to resolve this I had some ideas where I tried to add check to not update something twice (which is good still) but the problem was with an entity needing to also persist its relationships and the loop would never be solved or we'd end up with only a partial (and unpredictable) persist of the object graph. So I thought about it pretty hard and came up with what I think is the solution.

So the idea is basically like a depth first search. Only since we're using the metaphor of parent/child/sibling its more like a height first search. The problem is that this isn't exatly a strict tree, the relationships are seemingly random and branches can go from any node to any other node however two nodes cannot be parents of each other or children of each other. Furthermore, you cannot make any assumptions about which node begins the persist! So the rules I came up with are pretty simple:

Always persist upwards first.
Persist yourself.
If being persisted from above the persist downwards.

That's it essentially. So what happens is that the first nod persists its parents then they persist their parents etc. all the way to the top. Then since they were persisted from below and not from above they do not cascade persists downwards. It will wind back down to the original node then begin going downwards. Each downward node will persist all parents and then go further down the tree.

In this scenario you may have some nodes that will not get persisted (children of parents who do not have any other relations with the original persister), however this is predictable. Meaning, that when you do a persist you can readily know ahead of time what will not get persisted due to the relationships then call a persist on the other entities as needed. So I went from an endless loop to persisting 1720 entities in about .3 seconds by implementing those rules. So that made me happy.

Though now that I think more about it I'm not sure I handled it correctly for the sibling relationship scenario. It should be possible I just don't think I hooked it up yet. I'll take a look at it tonight.

Getting ready for a Beta 1

2006-11-13T14:07:00.000-08:00

I'm in the process of tidying up some things in preperation for a beta 1 build. I finally have the compiler and templates producing a dll that builds and "works". I'm sure there are bugs and issues still but for the most part it creates all of the data access and relationships and access / authorization / validation rules properly. I'm working on the Example website right now to try and get that a bit cleaner. I'm writing up an install page that will add the schema and default test data to a database.

I'm trying to decide whether or not to add the "action" syntax into the language before this next release. I've also been discussing things a little with others and I'm thinking about modifying the syntax to group validation/access rules in with fields. For example:

Currently the syntax looks like this:


family Test
{
    entity Blah
    {
        field string value;

        access deny * set value;
        validate value maxlength 64;
        validate value regex "\d{3}-\d{5}";
    }
}

But I'm thinking about grouping the rules into something a bit more logical so it might look more like this:


family Test
{
    entity Blah
    {
        field string value[9] = "\d{3}-\d{5}"
       {
           access deny * set;
       }
    }
}

I've also been thinking about more logical ways to specify what types of Entities you might want, such as readonly, info, etc. I'm thinking beyond just declaring an "Entity" you could declare a readonly entity or an info entity or something else perhaps. Also it might be good to be able to declare an "expression" which could be a reusable regular expression rule that you could use for validation rules on multiple entities. Also for custom rules I'm thinking some syntax such as this:


validate value Blah.Test.MyCustomRule in Blah.Test.dll

Which might require some way to add assembly references then call into the custom .dll to run the specified rule and you could put optional arguments into a dictionary and pass that in perhaps, or something like that.

So the big thing right now is trying to balance adding these features with getting a release out the door. I don't really have any other developers contributing here so getting anything done is not real fast but I would like to produce something not only useable but something that is actually beneficial! So the question is whcih features do I add in? Right now I'm shooting for only what I have done so far plus (perhaps) the action syntax. Then evaluate which features to add in for subsequent versions.

If anyone has any ideas or questions (or anything!) about entity languages and would like to discuss this please feel free to join the google group and start up a discussion or add an entry to the codeplex forum. I'm really ready for some feedback / critiques and some thought provoking discussion to help me get some of these ideas straight. Also if you think you could use something like this for an application of your own let me know because I'm looking for some contributors for sure.

2006-10-12T22:17:00.000-07:00

I'm making some good progress I believe. The project is looking good and I'm still very motivated to continue onward. The last couple days have been pretty interesting, 3 things have really happened.

The first thing that happened, which is very tangible, was that I moved the source code away from the Google code Subversion repository and into a project site hosted at Code Plex. I'm thinking this is a good move so far. The issue management and integrated discussion capabilities is FAR superior. Not to mention the Team Explorer integrated source control software is pretty sweet as well. I'm digging the free wiki pages built in and built in release storage locations. I'm very happy for this upgrade so far.

The second thing is that I watched a very interesting video about the SubSonic "Actionpack" the other day, which is a very interesting library that is similar in purpose to NBusiness and got some good ideas. For one thing the Build Provider section was way cool though it seems like it could be trouble to tie your build proccess to a database too extensively. However I must say I LOVED the "Query" class structure they built. I'm definitely thinking that we should so something like that as well. It would be a nice touch to allow a developer to add custom queries so easily.

The third thing that happened was that I have been talking to a friend of mine (Beau) about this project and he has always been especially interested in Data Access Code and database related things, after watching the Actionpack video he seemed pretty pumped to help me work on some of this stuff. He seemed to be happy to do it because he himself would like to use that code for some projects of his own and maybe at work. So finally getting some help would be great! I'd love some extra feedback for one thing but I'd also like to make some progress faster. This project is coming along nicely but its also got a lot of work left.

Status updates:
- esc is building and compiling output that builds
- NBusiness is templatized and Providerized
- demo is gone and is replaced with Example
- Example is a web app and an NUnit test project
- Installer will install, esc and NBusiness.dll into GAC

First post!

2006-08-17T18:17:00.000-07:00

I've hit a sort of milestone today. I have end to end generation happening (as imperfect as it is) and it seems to be working quite well. I made some tough decisions today and trimmed quite a bit of fat but for the better I believe. The code is looking pretty good and I already have lots of ideas how to make it better. I also packaged everything you would need to start generating entities into a single demo zipfile here:

http://nbusiness.googlecode.com/svn/trunk/demo/NBusinessDemo.zip

The most important milestone of today, though, is probably solidly defining how relationships work. I started by really breaking down all of the possible relationship types, here is what I came up with:

1:M (one to many)
M:1 (many to one)
M:M (many to many)
1:1 (one to one)

After thinking about these relationships a bit I wrote up these little diagrams:

Basically this shows that the 1:M and the M:1 relationships are the same but just from different perspectives. In these relationships you have a Parent and a Child. The parent can have multiple children but the Children can only have 1 parent. So if you have two entities, A{aid} and B{bid, aid} you can establish a 1:M relationship with this syntax on A:

relationship BList with B on aid as child;

This establishes a child relationship called BList with B using the aid field to link them. And now for the M:1 on B:

relationship A with A on aid as parent;

Now the M:M relationship is significantly different. The idea here is that A can have many B's and B can have many A's. In this scenario neither one can be considered a parent or a child so instead we use the term 'sibling'. So if we have entities A{aid} and B{bid} then we can create a M:M relationship with this syntax:

on A: "relationship BList with B as sibling;"

on B: "relationship AList with A as sibling;"

Its important that both entities have the sibling relationship in this case and that they do not declare which fields to link on. In this case the Compiler will generate an internal intermediate type known as AB with fields for each id field in both entities. When adding an instance of B to the BList on A you will actually be adding an instance of AB but this is transparent to the developer.

It's also interesting to note that the 1:1 relationship never really comes up... Or at least this is what I'm thinking at this point. Typically in order to establish a 1:1 relationship at least one of the entities would have to have a field that corresponds to another entity but if only one has the field then that's just a 1:M, if they both have id's for each other then you could have a true 1:1 but that begs to have the question "why bother?" asked. If you had a 1:1 entity relationship you might as well just consider combining the two entities into 1. Consider this 1:1 relationship:

A{aid,bid,c} , B{bid,aid,d}

How is this actually different than just having:

A{aid,bid,c,d}

Its really not, therefore a 1:1 is useless and should be avoided and is intentionally left out of the semantics of E#. If I'm making a huge logic flaw here someone please let me know!

Anyrate I'm onto working on some better debugging for the compiler and then implementing the output of the relationships!