Overview
Encountered
yet again; a legacy database, with no foreign keys, hundreds of tables, that formed the backbone of a REST API (CRUD style), supporting expansions and the like.
It wasn't that there were not identifiable relationships between objects, just the the way the database was generated meant that they were not captured in the database schema. But expansions had to be supported in an OData like fashion. So, assuming you had a resource collection called Blogs, and each Blog object had sub resources of Author and Readers, you should be able to issue a request like the following for a Blog with an id of 17:
http://..../api/Blogs/17?expand=Author,Readers
and expect a response to include expansions for Author and Readers.
That's easy then. Just use entity framework mapping/fluent API to configure an independent association with any referenced objects. Well, that can work and often does. But it does not cope well when some selected OData abstractions are included in the mix - and I was using these to expose some required filtering capabilities to consumers of the REST API. Simply put, when you were creating an ODataQueryContext using the ODataConventionModelBuilder type, independent associations cause it to implode in a most unpleasant fashion.
So, if I can't use independent associations, and each resource may have 1..n associations which are realised using joins, I can:
- Write a mega query that always returns everything for a resource, all expansions included
- Write specific queries by hand for performance reasons, as the need arises
- Generate code that map expansions to specific implementations
Writing by hand was going to be tedious, especially as some of the resources involved had 4 or more expansions.
When I thought about the possible expansions for a resource, and how they can be associated to that resource, using non FK joins, it became apparent that I was dealing with a power set of possibilities.
For the Blogs example, with expansions of Author and Readers, I'd have the power set:
{ {}, {Author}, {Readers} {Author, Readers} }
So the idea that formed was:
- Use a mini DSL to capture, for a resource, its base database query, and how to provision any expansions
- Process that DSL to generate pro forma implementations
- Use a T4 text template to generate C# code
I solved this in one way for a client, but then completely rewrote it at home because I thought it may be of use generally. I then extended that with a T4 template that generates an MVC 6 REST API...meaning all the common patterns that you typically see in a REST API were represented.
The actual VS 2015 C# solution is on
Github. The implementation itself is a bit more sophisticated than described here, for reasons of brevity.
DSL
The purpose of the DSL input file is to describe resources, their associated database base query, and any expansions and how they are realised. There are two formats supported - plain text and JSON.
An elided text file example for Blogs is:
1: tag=Blogs
2: singular-tag=Blog
3: model=Blog
4: # API usage
5: restResourceIdProperty=BlogId
6: restResourceIdPropertyType=int
7: #
8: baseQuery=
9: (await ctx.Blogs
10: .AsNoTracking()
11: .Where(expr)
12: .Select(b => new { Blog = b })
13: {joins}
14: {extraWhere}
15: .OrderBy(a => a.Blog.BlogId)
16: .Skip(skip)
17: .Take(top)
18: .ToListAsync())
19: #
20: expansion=Posts
21: IEnumerable<Post>
22: .GroupJoin(ctx.Posts, a => a.Blog.NonKeyField, post => post.NonKeyField, {selector})
23: #
24: expansion=Readers
25: IEnumerable<Party>
26: .GroupJoin(ctx.Parties.Where(r => r.Disposition == "reader"),
27: a => a.Blog.NonKeyField, party => party.NonKeyField, {selector})
28: #
29: expansion=Author
30: Party
31: .Join(ctx.Parties, a => a.Blog.NonKeyField, party => party.NonKeyField, {selector})
32: .Where(p => p.Author.Disposition == "author")
Relevant lines:
- Line 1: starts a resource definition
- Lines 5-6: allow this DSL instance to be used to generate a REST API
- Lines 8-18: The base query to find blogs, along with specific markup that will be changed the DSL processor (e.g. {selector}, {joins} and so on)
- Lines 24-27: A definition of an expansion - linking a reader to a blog if a Party entity has a disposition of "reader" and the column "NonKeyField" of a Blog object matches the same column in a Party object. The expansion results in an IEnumerable<Party> object.
- Lines 29-32: an Author expansion, this time (line 32) including a predicate to apply
Example class
After running the T4 template over the DSL file, a c# file is produced that includes a number of classes that implement the intent of the DSL instance.
The Blogs class (as generated) starts like this:
1: public partial class BlogsQueryHandler : BaseQueryHandling {
2:
3: protected override string TagName { get; } = "Blogs";
4:
5: public const string ExpandPosts = "Posts";
6: public const string ExpandAuthor = "Author";
7: public const string ExpandReaders = "Readers";
8:
9: public override IEnumerable<string> SupportedExpansions
10: { get; } = new [] { "Posts", "Author", "Readers"};
11:
Points:
- Line 1: The Blogs query handler class subtypes a base type generated in the T4 that provides some common to be inherited behaviour for all generated classes
- Lines 5-7: All the expansions defined in the DSL instance are exposed
- Lines 9-10: An enumerable of all supported expansions is likewise created
Example method
Harking back to the power set comment, a method is generated for each of the sub sets of the power set that represents the query necessary to realize the intent of the expansion (or lack thereof).
Part of pre-T4 activity generates queries for each sub set using the content of the DSL instance. Methods are named accordingly (there are a number of configuration options in the T4 file, I'm showing the default options at work).
As below, the method name generated in T4 for getting blogs with the Author expansion applied is Get_Blogs_Author (and similarly, Get_Blogs, Get_Blogs_Readers, Get_Blogs_Author_Readers).
1: private async Task<IEnumerable<CompositeBlog>>
2: Get_Blogs_Author(
3: BloggingContext ctx,
4: Expression<Func<Blog, bool>> expr,
5: int top,
6: int skip) {
7: return
8: (await ctx.Blogs
9: .AsNoTracking()
10: .Where(expr)
11: .Select(obj => new { Blog = obj })
12: .Join(ctx.Parties,
13: a => a.Blog.NonKeyField,
14: party => party.NonKeyField,
15: (a, author) => new { a.Blog, Author = author})
16: .Where(p => p.Author.Disposition == "author")
17: .OrderBy(a => a.Blog.BlogId)
18: .Skip(skip)
19: .Take(top)
20: .ToListAsync())
21: .Select(a => CompositeBlog.Accept(a.Blog, author: a.Author));
22: }
Some comments:
- Line 1: Declared privately as more general methods will use the implementation
- Line 3: The EF context type is part of T4 options configuration
- Line 4: Any 'root' resource expression to be applied
- Lines 5-6: Any paging options supplied externally
- Lines 7-25: The generated query, returning an enumerable of CompositeBlog, a class generated by DSL processing, that can hold the results of expansions and the root object
Generated 'top level' methods
As the generated 'expanded' methods are declared privately, I expose 'top level' methods. This makes the use of the generated class easier, since you pass in the expansions to use, and reflection is used to locate the appropriate implementation to invoke.
Two variants are generated per resource class - one for a collection of resources, one for a specific resource. The 'collection' style entry point is:
1: public async Task<IEnumerable<CompositeBlog>>
2: GetBlogsWithExpansion(
3: BloggingContext ctx,
4: Expression<Func<Blog, bool>> expr = null,
5: int top = 10,
6: int skip = 0,
7: IEnumerable<string> expansions = null) {
8: return await GetMultipleObjectsWithExpansion<CompositeBlog, Blog>
9: (ctx, expr, expansions, top, skip);
10: }
11:
12:
Comments:
- Lines 3-7: The EF context to use, along with a base expression (expr) and paging requirements and any expansions to be applied
- Lines 8-9: Call a method defined in the BaseQueryHandler generated class to find the correct implementation and execute
Example use
Imagine this closely connected to a REST API surface (there is a
T4 template that can do this, that integrates with
Swashbuckle as well). The paging, expansions and filter (expression) requirements will passed in with a request from an API consumer, and after being sanitised, will be in turn given to a generated query handler class. So the example given is what one might call contrived.
A
concrete test example appears below:
1: using (BloggingContext ctx = new BloggingContext()) {
2: var handler = new BlogsQueryHandler();
3: var result = await handler.GetBlogsWithExpansion(
4: ctx,
5: b => b.BlogId > 100,
6: 10,
7: 10,
8: BlogsQueryHandler.ExpandAuthor,
9: BlogsQueryHandler.ExpandReaders);
10: // .. Do something with the result
11: }
Comments:
- Line 1: Create a context to use
- Line 2: Create an instance of the generated class
- Line 3: Call the collection entry point of the generated class
- Lines 4-7: Supply the EF context, an expression and top and skip specifications
- Lines 8-9: Add in some expansions
Customisation
The T4 template has a 'header' section that allows for various options to be changed. I won't go into detail, but it is possible to change the base namespace for generated classes, the EF context type needs to be correct, whether a JSON or text format DSL file is being used, whether the 'advanced' DSL form is used - and so on. The
GitHub page supplies more detail.
// ****** Options for generation ******
// Namespaces to include (for EF model and so on)
var includeNamespaces = new List<string> { "EF.Model" };
// The type of the EF context
var contextType = "BloggingContext";
// Base namespace for all generated objects
var baseNamespace = "Complex.Omnibus.Autogenerated.ExpansionHandling";
// The DSL instance file extension of interest (txt or json)
var srcFormat = "json";
// True i the advanced form of a DSL instance template should be used
var useAdvancedFormDSL = true;
// Form the dsl instance file name to use
var dslFile = "dsl-instance" + (useAdvancedFormDSL ? "-advanced" : string.Empty) + ".";
// Default top if none supplied
var defaultTop = 10;
// Default skip if none supplied
var defaultSkip = 0;
// True if the expansions passed in shold be checked
var checkExpansions = true;
// If true, then expansions should be title cased e.g. posts should be Posts, readers should be Readers and so on
var expansionsAreTitleCased = true;
// ****** Options for generation ******
