Support for TextTransform.exe in T4 Toolbox

Coordinator
Apr 2, 2009 at 12:54 PM
T4 toolbox currently supports code generation only at design time, inside of Visual Studio. We are currently considering implementing support for code generation with TextTransform.exe (in command line mode, such as at build time) and I would like to hear everyones thoughts on this.

* How important is the ability to generate code using TextTransform.exe?
* Would you like for T4 Toolbox to support TextTransform.exe?
If you do:
* How do you envision adding generated files to their respective projects for compilation?
* Do you care about adding generated files to source control when in command line mode?

Thanks,
Oleg
Apr 2, 2009 at 2:11 PM
Very important.
In a build script you may put a dependency between items.

Giulio
Apr 2, 2009 at 2:24 PM
Very very important here as well.  Being able to run it from a command prompt would open a whole new way for T4 Toolbox to be used.  It's quite non-traditional to use the save-file method as the only means to kick off the generation process.  With TextTransform.exe support, we'd be able to create GUI utilities to collect inputs from the user and run the generation process, accordingly.

WRT putting files into their projects and adding files to source code, I don't think I would do it much differently than how I'm doing it in S#arp Architecture.

Billy
Coordinator
Apr 2, 2009 at 3:16 PM
Guilio, could you elaborate?
Billy, could you describe how you are doing it now in S#arp Architecture?

Is there a way to manipulate Visual Studio projects directly? (Aside from generating entire project files as text) Are there MSBuild APIs for that?

Oleg
Apr 2, 2009 at 4:51 PM
Since I'm at least partially responsible for triggering this discussion, let me add my two cents.

I recently started looking at using the T4 engine to replace several custom-coded internal code generators at work.  I came across the T4 Toolbox while Google'ing and downloaded it to try.  Like others have also found, the current release of the Toolbox doesn't work with the command line TextTransform.exe.

Our intended usage is to generate complete, build-able projects, including all source files and the .csproj (standard msbuild projects not full VS C# projects), through a collection of T4 templates.  Both the code generation and the building of the generated projects will be done by an automated build process on machines that do not have Visual Studio installed.  The generated files will not be source-controlled, only the .tt templates.

To that end, I downloaded the T4 Toolbox source code, extracted several classes from the DLL to .tt files and removed all references to the VS automation objects.  Additionally, I had to rewrite the OutputManager.AppendFiles() method to create the files on disk instead of delegating to Oleg's custom directive processor.  If anyone wants the source, I'd be happy to send it on.

I would like to see the strategy for managing the creation of output files refactored so that the author of the code generator is able to choose VS integration or not, depending on the situation.

Dylan Bourque
Coordinator
Apr 7, 2009 at 1:11 PM
Thanks Dylan. I agree that creation of output files needs to be separated from code generation itself and encapsulated in a strategry that can be replaced at run time depending on what the author needs. Using OutputManager as the base strategy is an option, however, it doesn't solve the larger issue, which is interface and implementation dependencies on Visual Studio automation model (EnvDTE and other assemblies) in the TransformationContext class. Some code generators may require access to Visual Studio automation model, while others may not care about it.

With this in mind, I would like to suggest keeping OutputManager as is (mostly a data transfer object). As an alternative, I would like to suggest introducing two additional classes: VisualStudioContext and CommandLineContext. Each class would be responsible for a) implementing host-specific output file management and b) providing access to host-specific functionality, such as EnvDTE.Project, EnvDTE.ProjectItem currently exposed by TransformationContext. The author would choose which context/strategy to use by including an appropriate file:

<#@ template language="C#" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\VisualStudioContext.tt" #>
<#
    this.WriteLine(VisualStudioContext.Project.Name);
#>

or

<#@ template language="C#" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\CommandLineContext.tt" #>
<#
    this.WriteLine("No Access to Visual Studio Here");
#>

The different contexts would register themselves as handlers for the TransformationContext.TransformationEnded event and save output files in a mannger appropriate to the host. VisualStudioContext would reference the automation assembliesand use EnvDTE objects to add output files to target projects and source control. CommandLineContext would use MSBuild APIs to add output files to target projects only. This way the core output management functionality would be consistent between the two contexts, the authors would be able to choose which context they want their code generator to run in, and if they choose CommandLineContext, there are no EnvDTE dependencies compiled into the template.

Thoughts?
Apr 7, 2009 at 4:17 PM
Sorry, I've been away for a couple of days.

I've in mind this scenario: generating code through T4 based on some source / "metadata" file that may frequently change. For example, the metadata represents a WCF service interface from which I generate different code snippets: test code, wrapper code, mock code, setup code, all of which depend on the interface definition.
As it is currently, if use your toolbox in my templates, I cannot use an MSBuild task (calling TextTemplate) to regenerate the code when the file containing the metadata changes (while it could be really useful).

I think it would be much better to encapsulate, as much as possible, in the toolbox the differences between the two contexts: in my scenario both a developer and a central build needs to regenerate code.

Hope this clarifies.
Coordinator
Apr 7, 2009 at 5:45 PM
So, is this a third scenario - build-time code generation in Visual Studio? What would you expect to happen with output files generated this way if solution is under source control? Would you expect new output files to automatically appear in the solution?
Apr 8, 2009 at 9:05 AM

Not necessarily: integration with Source Control is natural in “interactive mode”, i.e. when I add/edit a T4 file in VS. In a “batch mode” it’s an hindrance having generated files under SCM: they are more like binaries. What must be easy is included newly generated code in the compile phase, but this not hard tweaking the MSbuild script.

If think of this as an advanced scenario for heavy usage, not the casual template.

Hope this clarifies a bit.

Coordinator
Apr 8, 2009 at 12:22 PM
Thanks for the clarification, Giulio. I'm glad we are down to just two scenarios - design-time code generation, which needs to support source control, and build-time code generation that doesn't. Have you had a chance to try version 9.3 of the T4 Toolbox? It provides support for automatically adding output files to Visual Studio projects and allows you to write code like this:

<#
  MyTemplate template = new MyTemplate();
  template.Output.File = "<Path>\Output.cs";
  template.Output.Project = "<Path>\Target.csproj";
  template.Render();
#>

which will save template output to a file called Output.txt and automatically add it to a project called Target.csproj.

When this template is running in command line host, TextTransform.exe, would you expect it to work the same? In other words, would you expect T4 Toolbox to take care of adding the generated file to the target project for compilation, or do you need to control this explicitly, say in a MSBuild script?

Oleg
Apr 9, 2009 at 5:21 PM
Oleg,
Your plan sounds great.  Like I said in my previous post, I've gotten past my immediate issue and have gone on to proof-of-concept stage.  I'd be happy to discuss technical details of this new implementation.  Don't know if I'll have any time to devote to working on it though.  I have a 2-month-old baby and he takes priority. ;-)

Giulio,
I'm working on pretty much the same scenario as you.  We have a set of metadata files (XML in our case) that define the code to be generated and we'll need to be able to invoke the T4 engine when needed to (re)generate the projects on a dedicated build machine without Visual Studio installed.  My plan is to build up a set of custom MSBuild targets with the project metadata XML file as its input and the generated files as outputs.  For our system, we're also generating MSBuild project files and building the generated projects as part of our automated build.
Coordinator
Apr 11, 2009 at 6:08 PM
Edited Apr 11, 2009 at 6:09 PM
Thanks Dylan. I completely understand, I'm not working on this full time myself :). I will write up the details in an enhancement request. This is an important feature. I hope someone will be interested and have time to incorporate it in the core framework.
Apr 20, 2009 at 1:26 PM
I just spent a while trying to get it working with TextTransform.exe before I realised that this is not a trivial matter, and then ended up here.

We have a similar requirement here to what has already been stated: automated T4 code generation on the build server (based on template files in source control). The output files do not need to be checked in as they are generated on every build.
Coordinator
Apr 20, 2009 at 2:02 PM
Here is the enhancement request I created for this: http://t4toolbox.codeplex.com/WorkItem/View.aspx?WorkItemId=12979. I agree, this is not a trivial change, but shouldn't be too difficult to implement.
Developer
Apr 21, 2009 at 10:35 AM
Hi guys,

I was collaborated with Oleg with the TextTransform.exe command line.

My scenario is the same as wmccafferty said "With TextTransform.exe support, we'd be able to create GUI utilities to collect inputs from the user and run the generation process, accordingly.", then I would have a GUI for get by sample the Datamodel and then create an entiere VS solution thare are ready to compile. 

Of course I'll need access to VS automatation for add items to projects.

Other stuff that I told to Oleg was the possibility to have a wrapper to communicate with MyGeneration templates, because there a lot of templates many of theses for DataBase scripting, and will be great to reuse it.


Coordinator
Apr 21, 2009 at 11:29 AM
Marcos, if we can use MSBuild API instead of Visual Studio automation model to add items to projects, would you still need to have Visual Studio running in your scenario?
Apr 23, 2009 at 10:13 AM
Edited Apr 23, 2009 at 10:14 AM

Hi,could somebody provide status?

I'm currently starting using T4/T4 Toolbox  in big project.

We we need generate DAL from the database during compile time.

Thanks.

Coordinator
Apr 23, 2009 at 12:05 PM
Sure - it's on the to-do list :). Want to help?

Oleg
Apr 29, 2009 at 8:53 AM
Edited Apr 29, 2009 at 8:54 AM
I'm interested too. We want to generate C++ code for automated builds (makefiles/jam).

Also I wrote the TextTransform.rules file for Visual Studio which supports generating of code using TextTransform.exe.
Grab it here as an example. It relies on .ttt extension.
Apr 30, 2009 at 5:26 PM

Yes, how can I contact youto discuss details?

May 7, 2009 at 6:51 PM
I am interested in this feature as well. Would love to help with implementation if needed.
Coordinator
May 7, 2009 at 11:26 PM
It sounds like we have a few people interested in implementing this - lagear, tom3m, loyso, crmesser. Have you guys had a chance to take a look at work item 12979? Do you have any feedback?

If you agree with the design, I think the best way to tackle this is for someone to take the lead and implement it. You can submit your implementation as a patch on the Source Code page. The rest of us can download it and to help with the code reviews and testing

I will definitely help with code reviews and testing (this needs to work well with the rest of the framework). If you have questions, I'll be happy to answer them here. However, I don't think I'm the right person to be the lead on this feature - I have neither bandwidth nor plans to use this feature in the near future. What do you think?

Oleg
May 8, 2009 at 4:40 PM

This is probably out of the scope of this Issue, but definitly related. It would be nice if the framework provided a default Command Line Host so you didn't have to use TextTransorm.exe. Microsoft has an example of a custom host here but it isn't much use without some more robust logic (in fact it doesn't even completely implement the ITextTemplatingEngineHost interface). I was able to leverage this and a modified (read: hacked) T4Toolbox framework to work as a standalone class library assembly. This allowed me the create a console application to parse input/output parameters and pass them to properties in my custom host and access them via reflection in my generator template similar to what is shown here but cleaner bc the parsing of parameters is encapsulated in my console application's code.

Coordinator
May 8, 2009 at 9:55 PM

I'm not sure I would want to create another T4 host. We already have 4 different hosts from Microsoft - Visual Studio, TextTransform.exe, GAX and ASP.NET MVC. Each comes with its own implementation, subtle and not so subtle differences, making sharing all but trivial templates impossible.

Have you considered packaging the UI logic to capture input parameters in a Visual Studio project item wizard and using the standard ITextTemplating service to generate code with T4 in Visual Studio?

Jun 29, 2009 at 7:44 AM

There are a couple of strange commits from lagear:

VisualStudioContext.tt:

/// Provides a Visual Studio instace when <see cref="TransformationContext"/> is running in command line T4 host (TextTransform.exe)

The code runs VS instance in background. As I understand, it's definitely different from the proposed CommandLineContext.tt concept, isn't it?

 

Coordinator
Jul 7, 2009 at 1:52 PM

This code is one of the early attempts to implement support for TextTransform.exe host that Marcos and I worked on. I pinged him to see if he wants to work on the changes we discussed in this thread. Would you be interested in helping with this?

Developer
Jul 7, 2009 at 3:28 PM

loyso,

 

The first step to run t4 Toolbox was run from command line, but there was a problem with the IServiceProvider, and the current code expected a VisualStudio object, so I most to load this Visual Studio in background.


Also this Visual Studio object let to us to Add, Delete, Move, etc intro a VS project.'

 

Regards,

Oct 5, 2010 at 7:08 PM
lagear wrote:

My scenario is the same as wmccafferty said "With TextTransform.exe support, we'd be able to create GUI utilities to collect inputs from the user and run the generation process, accordingly.", then I would have a GUI for get by sample the Datamodel and then create an entiere VS solution thare are ready to compile. 

Of course I'll need access to VS automatation for add items to projects.


Hi Lagear,

Did u manage to achive this using T4 Toolbox ?

I want similar kind of thing , in which I could send parameters to my Template, which I have collected from end user using Wizard approach.

Any help will be much appreciated .

thanks in advance,

SWSF.

Coordinator
Oct 5, 2010 at 7:58 PM

Implementing support for TextTransform.exe is not trivial. An early attempt to do this didn't go very far and chances of someone volunteering to implement this are probably not very high. MSBuild integration and pre-processed templates are more viable than TextTransform.exe today.

Oleg

Oct 6, 2010 at 6:37 AM

Hi Oleg ,

So then can u plz help me here, I want to pass parameters to my Template at runtime and want to execute those tTemplates runtime too in my WinForm application.

How can I do this ???

Thanks,

SWSF

Oct 13, 2010 at 8:02 AM

What exactly is the status of MSBuild integration and what needs to be done to implement it? I am completely unfamiliar with the internals of T4 Toolbox but I would like to understand the effort required to implement this - the functionality is fairly useful for me, so if the effort is reasonable, I might be able to fit it into the budget and implement this functionality.