how to create an output file only if it does not exist

Aug 7, 2009 at 7:59 PM
Edited Aug 7, 2009 at 8:03 PM

All --

I need to know how to create an output file only if it does not exist.

How can it be done?

 

I see there is a patch uploaded here...

http://t4toolbox.codeplex.com/SourceControl/PatchList.aspx

...which seems to address the matter but it appear to have not been integrated.

 

Note that I did try to do this the GEFN way.

I simply passed-in the output-path as a parameter in my template set.

Next I did a call to "File.Exists".

Curiously, the file is not overwritten but if the file aready exists then the Transform process deletes the file. Ug.

(It has strawberry and vanilla but I want chocolate.)

:-(

My failed attempt is excerpted below, just in case I missed something and have a coding error and it should work but for my error.

 

//This will try to pass in the PathToOutput (or can it be got programmatically at template transform-time?) and try to 

myFileNameTemp = 
	this.ClassPrefixToUseForOutput + myObjectNameCleanedTemp + this.ClassSuffixToUseForOutput + ".cs";

myFilePathCompleteTemp = 
	Path.Combine(this.PathToOutput, myFileNameTemp);

//NOTE. 20090807. Old. This works in that it writes an output file but it overwrites the file if it already exists.
//this.DefaultTemplateToUse.RenderToFile(myFileNameTemp);

//NOTE. 20090807. New. This does not work. This deletes the file if it already exists and does not write a new one.
if (File.Exists(myFilePathCompleteTemp))
{
	//Continue.
}
else
{
	//this.DefaultTemplateToUse.RenderToFile(myFileNameTemp);
}

Why do I want to do this? Well, I have a common pattern of generating, which I have seen others do and which works for me. Generate a base class with core functionality. Create a manual derived class, which inherits from the base class, to put extensions to the base, etc.  As such, I generate a never-manually-touch output file as a base class which gets re-generated over and over as needed and as it develops AND I like to "generate the derived class if it does not exist" to give the project a jump start and eliminate some one-time file creation for the hand-written class. Etc. That is my motivation.

What do you think?

Please advise.

Thank you.

-- Mark Kamoski

Coordinator
Aug 11, 2009 at 10:58 PM

I was reluctant to implement this in the core framework when the patch was initially submitted because I don't think it's a good idea to generate "empty" output files, files that serve as a "starting" point for developers to finish . Having these output files seems convenient when I'm building a code generator. However, once the code generator is ready and I start using it to generate numerous source files, having an empty ouput file per generated class quickly clutters the project. Creating these partial classes by hand is quick enough, so I tend to err on the side of long-term simplicity and don't generate them.

However, this request keeps coming up. If you are interested in implementing it, I can help you extend the core framework of T4 Toolbox to support this feature as a Template.Output option.

Aug 13, 2009 at 4:40 PM
Oleg --

Regarding this...

>>>...I don't think it's a good idea to generate "empty" output files, files that serve as a "starting" point for developers to finish . Having these output files seems convenient when I'm building a code generator. However, once the code generator is ready and I start using it to generate numerous source files, having an empty output file per generated class quickly clutters the project. Creating these partial classes by hand is quick enough, so I tend to err on the side of long-term simplicity and don't generate them.

...I do tend to agree, in general, however, it would still be good to know how to avoid overwrite, maybe.

Regarding this...

>>>However, this request keeps coming up. If you are interested in implementing it, I can help you extend the core framework of T4 Toolbox to support this feature as a Template.Output option.

...I might have some moments to spend on extending the core framework...

...but...

...what I had in mind is, I think, a bit more simple and it is template-based... that is, in my project-specific T4-file I want to set a property "OverwriteExistingOutput" and then in the shared-T4-Generator at template-run-time I want to simply check if the file exists, maybe something like this...


myDatabase.Refresh();
string myDatabaseObjectNameActualTemp = "";
string myDatabaseObjectNameCleanedTemp = "";
string myOutputFileNameTemp = "";
string myOutputFilePathTemp = "";
string myOutputFilePathComplete = "";

foreach (Microsoft.SqlServer.Management.Smo.Table myDatabaseObjectTemp in myDatabase.Tables)
{
if (myDatabaseObjectTemp.IsSystemObject)
{
//Continue.
}
else
{
myDatabaseObjectNameActualTemp = myDatabaseObjectTemp.Name;
myDatabaseObjectNameCleanedTemp = this.GetDatabaseObjectNameCleaned(myDatabaseObjectTemp.Name);

this.DefaultTemplateToUse.DatabaseObjectNameActual = myDatabaseObjectNameActualTemp;
this.DefaultTemplateToUse.DatabaseObjectNameCleaned = myDatabaseObjectNameCleanedTemp;
this.DefaultTemplateToUse.DatabaseObjectNamePluralized = this.GetDatabaseObjectNamePluralized(myDatabaseObjectNameCleanedTemp);

this.DefaultTemplateToUse.PrimaryKeyColumnConverter = this.GetPrimaryKeyColumnConverter(myDatabaseObjectTemp);
this.DefaultTemplateToUse.PrimaryKeyColumnDefault = this.GetPrimaryKeyColumnDefault(myDatabaseObjectTemp);

this.DefaultTemplateToUse.PrimaryKeyColumnName = this.GetPrimaryKeyColumnName(myDatabaseObjectTemp);
this.DefaultTemplateToUse.PrimaryKeyColumnType = this.GetPrimaryKeyColumnType(myDatabaseObjectTemp);

ArrayList myUniqueKeyColumnList = this.GetUniqueKeyColumnList(myDatabaseObjectTemp);
this.DefaultTemplateToUse.UniqueKeyColumnList = myUniqueKeyColumnList;

myOutputFileNameTemp = this.ClassPrefixToUseForOutput + myDatabaseObjectNameCleanedTemp + this.ClassSuffixToUseForOutput + ".cs";
myOutputFilePathTemp = this.GetPathToOutputFile();
myOutputFilePathComplete = Path.Combine(myOutputFilePathTemp, myOutputFileNameTemp);

if (this.OverwriteExistingOutput)
{
this.DefaultTemplateToUse.RenderToFile(myOutputFileNameTemp);
else
{
if (!File.Exists(myOutputFilePathComplete))
{
this.DefaultTemplateToUse.RenderToFile(myOutputFileNameTemp);
}
}
}
}

...but I cannot seem to get that to work.

What do you think?

Please advise.

Thank you.

-- Mark Kamoski
Coordinator
Aug 13, 2009 at 5:46 PM

This code doesn't work because T4 toolbox assumes that every output file is generated every time. If a previously generated file is not regenerated, it assumes this file is no longer necessary and removes it. A change in the core output management functionality is necessary to make this work as you expect.

Aug 13, 2009 at 6:24 PM
Oleg --

Regarding this...

>>>This code doesn't work because T4 toolbox assumes that every output file is generated every time... A change in the core output management functionality is necessary to make this work as you expect

...that is fine...

...I will try to look at the source and help a bit, if possible.

Thank you.

-- Mark Kamoski
Aug 17, 2009 at 8:42 PM
Edited Aug 17, 2009 at 8:56 PM
olegsych wrote:

....A change in the core output management functionality is necessary to make this work as you expect.

Oleg --

Unfortunately, my attempts at making the change have failed.

My goal was to add an overload.

The current method is... RenderToFile(string filename)

The overload would have been... RenderToFile(string filename, bool overwriteIfExisting)

I think I might have been a bit close; but, I could not propogate the flag into the right places and so the transform-time processing always deleted it (though did not regenerate it if the overwriteExisting=false). I think there is a pre-transform step or something I missed. I couldn't quite follow the call sites very well. My lack of prowess I'm afraid. I spent some hours on it with no luck so I have to move on. I did learn a few things. But, the solution was, I thought, a simple fix and due to my lack of skills it was taking longer than I thought it should so I abandoned the idea. As it is, as I have said, I see the skipIfExisting functionality as a minor one-time savings so I do not care too much about it. If I get a chance, I will try again. I think my problem, if you are interested, was that I thought you said to use the CopyToOutputDirectory enum and I did try that but since outputFiles is a collection it was not clear for me to find a way to peel the right outputFiles[n].CopyToOutputDirectory setting to pass it into the method call-chain to get the desired effect. Etc.

I uploaded my code in case you want to see it.

See http://t4toolbox.codeplex.com/SourceControl/PatchList.aspx which is annotated as follow...

3615... This is NOT a patch. This is only an upload for Oleg Sych. This is just in case he wants to see my failed attempt at making a change to add an option to SkipOverwriteIfExisting. The file name is "T4Toolbox_Skip_Overwrite_Existing_Failed_Change_200908171523.zip".

Thank you.

-- Mark Kamoski

Coordinator
Aug 18, 2009 at 11:56 PM

Mark,

Here is what I think we can do to implement this functionality.

* Modify OutputInfo.cs to add a new property called PreserveExisting of type bool to the OutputInfo class. This property will be false by default and can be set to true to a) preserve an existing output file with the same name during initial code generation b) prevent the output file from being deleted or overwritten during re-generation.

* Modify code in OutputManager.cs that accumulates generated content to make sure that PreserveExisting property flows from templates to output files.
- in Validate(output) method, throw exception when output.PreserveExisting = True for the standard output of the transformation
- in Validate(output, previousOutput) method, throw exception when previousOutput.PreserveExisting != output.PreserveExisting
- in AppendToOutputFile method, assign outputFile.PreserveExisting based on output.PreserveExisting

* Modify code in DteProcessor.cs that writes genereated content to output files. This code uses two mechanisms to keep track of previously generated files - the list of project items nested under the main .tt file and the code generation log file (.tt.log). The list of nested project items is used unless one or more output files are located in a different folder or a different project. If the output file is included in either of these two lists, the code written by hand may by the user could be automatically deleted if the particular template was not regenerated, so we need to prevent that.
- in CreateLogIfNecessary, skip the output file (prevent it from being written to the log) if output.PreserveExisting = True
- in UpdateOutputFile, skip updating output file if output.PreserveExisting = True
- in AddToSolution, add the output file to the project instead of the template item if it has PreserveExisting = True to make sure it is not deleted during re-generation.

Does this make sense?

Oleg

Coordinator
Aug 19, 2009 at 12:05 AM

On a second thought, we may be two separate scenarios:

- preserve existing files when output is generated

and

- overwrite the existing file when output is generated but don't delete it when it is no longer generated

Unless I'm mistaken, you have the first scenario, while Billy McCafferty has the second scenario in S#arp Architecture. We may need to have two separate properties, one to control whether existing file is overwritten during code generation (i.e. PreserveExisting) and another to control whether old output file is deleted when it is not regenerated (something like DeleteWhenNotRegenerated).

Oleg

Aug 19, 2009 at 1:33 PM
That makes good sense.

I am on vacation for a while but will look at it when I get back.

Time permitting.

Etc.
>
Aug 19, 2009 at 1:37 PM
Yes. Exactly. Both are needed. The first case is simple, I think, as I
had a hacked version doing just that. However, the second case is
harder, I think, because that is where I got stuck because it
seems that cleanup occured before generation.
>
Sep 2, 2009 at 1:32 PM
olegsych wrote:

...Here is what I think we can do to implement this functionality.... Does this make sense?

Oleg --

FYI, I have set my plans of implementing "create only if not existing" functionality aside, for several reasons.

(1). I switched from a generate-base-classes-approach to a generate-partial-classes-approach. I did this because in my case it yields the same result. It also eliminates the need to manage an inheritance-chain. Furthermore, it simplifies the code generation templates.

(2). Given (1), given YAGNI ( http://xp.c2.com/YouArentGonnaNeedIt.html ), the functionality should not be implemented now simply because I do not need it now.

(3). I have reconsidered your post above, which starts "I was reluctant to implement this", wherein you give some good reasons as to why why the functionality is not necessary. You are quite correct in many (perhaps all) of your points.

(4). I am a bit swamped right now.

Etc.

Note that that I do subscribe to this CodePlex project. As such, if I see significant outcry for the feature, then I might revisit this.

Note that if I do find time and if I can rebut the YAGNI argument, then I might revisit this.

Etc.

Thank you.

-- Mark Kamoski

Coordinator
Sep 2, 2009 at 5:35 PM

I understand. Thanks for sharing your thoughts on this.

Sep 3, 2009 at 5:46 AM

Hi Oleg,

Any chance of you (or someone) continuing to look at and perhaps implementing this feature, "preserve existing files when output is generated"? I could use it but don't really know where to start...

Coordinator
Sep 3, 2009 at 11:28 AM

Sure, we'll keep it on the list.

Nov 21, 2009 at 6:44 PM

Currently, T4 Toolbox assumes that all files are regenerated every time. If a particular file is not regenerated, it assumes that it is no longer necessary. Whether the Run method got call or not.

I think that T4 Toolbox should only delete files if Run() is called. Since we have a run method that must be explicitly call. I think it's a reasonable assumption that we should "do nothing" if Run didn't get call. Needless to say I was a little shock to find out my file disappear after I commented out the Run :)

 

My other Thought on this is instead of Run we should have Generate and ReGenerate

Generate() should Append to the current log. (Current we override the old log). Problem with overriding the old log is that if we generate a different set of file T4 Toolbox each time will not have a full list of files that got generated. This resulted in not every files will get removed when we want to remove them.

ReGenerate() should Overide the old log, since we will also delete all previously generated file and generate a new set of files.


Perhaps we should also expose Delete(logFIleName) to delete all file in a particular log file. I think all the necessary code is already there this is just a matter of semantic. Right now almost everything is automatic. The suggestion above should give the caller better control of how T4 Toolbox run.

 

 

Coordinator
Nov 22, 2009 at 1:58 PM
Edited Nov 22, 2009 at 2:09 PM

@firefly,

It sounds like you are trying to use T4 Toolbox to perform single time code gen, which is not what it was designed to do. Could you describe the scenario you are trying to implement in more detail? What are you generating the code from? What is the workflow?

Thanks,
Oleg

P.S. Here are other threads where we had a similar discussion

http://t4toolbox.codeplex.com/Thread/View.aspx?ThreadId=69836

Nov 22, 2009 at 7:02 PM

Oleg,

Actually I am doing selective code generation. A single time code gen is fairly simple really. Just remove the *.tt.log file and get it done with. My work flow is at follow.

I use code gen to generate all a rough foundation. Something similar to what Sharp Architecture is doing. Then I turn off the code generation and edit the file directly to get the customization that I want. Say that I want to customize the view I only customize one particular view file, not all. But this give me something to model my template from. While it's true that I can edit the template file directly its two step approach it make it painful to switch back and forth (missing a semi colon and so on...) then wait for it to generate all the file.

After I finished up with the customization of my site. I then goes back to customize the template base on my edited code. I just wrote a GUI that allow me to do this quickly. Code generation is tell to regenerate again to refresh all the set of file. This might goes on a few iteration and that's why I would like to turn on/off the code generation at will. I also would like to delete all the old file every times I tell it to regenerate to prevent any orphan class.

Best,

Hoang

Nov 22, 2009 at 7:55 PM
Consider generating a partial class. My Northwind01 sample in the
Patches section shows how.

Or consider generating a concrete base class.

Or consider generating an abstract base class.



On Nov 22, 2009, at 2:02 PM, "firefly4321"
<notifications@codeplex.com> wrote:
> From: firefly4321
>
> I use code gen to generate all a rough foundation. Something similar
> to what Sharp Architecture is doing. Then I turn off the code
> generation and edit the file directly to get the customization that
> I want
Nov 22, 2009 at 8:25 PM

Mark,

Thanks for the advise. I actually do have all three in my architecture but still there is a gap to be fill. For reason my last post didn't show up. Maybe I forgot to hit send :)). But a better terminology for my work flow would be this:

Say that I want to build a car. I would come to the molding guy. Pick from a set of molds and ask him to give me a skeleton of a car. I then customize that skeleton heavily to build the car. Once I am done I can goes back to the molding guy and say this is the car that I want can you make me a mold for it?

The reason for this work-flow is that it's much easier to build the mold (our template) from something concrete. In fact most molder will also need to build one sample of the product before they build the mold. The concrete and abstract base class is important for the application architecture as a whole especially if we need to change something later. The partial class is nice when we make minor tweak to our mold. But if the change is fairly large. It's always easier (for me anyway) to build the mold from an actual sample.

Hoang

Nov 23, 2009 at 2:55 AM
What works for you is best for you , I suppose.

I think my style is different than yours.

I think the following general rules are good for code generation.

Never edit generated code manually.

If you have to change the generated code, then do so by q editing the
template and regenerate.

Etc.


>
Nov 23, 2009 at 4:34 AM

Care to explain your work flow then? Please explain more than just using partial class or code abstraction. I am open to suggestion but I need something more concrete :)

I do agree that generated code shouldn't be edited. However I don't think I made myself clear here. Say I need to generate 100 similar classes. Doing a regen each time is costly and it's much harder to debug unless the tweaking is extremely minor.

So what I do is that I tweak one class directly to do what I want. Then I edit the template base on that model class so I can regen the 100 classes again. Eventually they all get regenerated automatically...

Nov 23, 2009 at 5:45 AM
I do the following.

Make template.

Generate code as partial classes in a "Generated" folder.

Add custom code to hand-maintained partial class in a different
physical location than the "Generated" folder.

Change templates as necessary.

Reuse templates on all projects.

Etc.

I uploaded a full application sample as "Northwind01_*" in the
"Patches" section so please do take a look.

HTH.
>
Coordinator
Nov 23, 2009 at 12:40 PM

Hoang,

If I understand your description, the T4 Toolbox already works this way. It will generate the initial set of files, which you can modify, make them work. As long as you don't run the code generator again, these files will not be overwritten. Once you have worked out one concrete implementation, you can go back, modify the templates and regenerate all output files based on the new implementation and remove all files that are no longer necessary. Is this what you are trying to do?

Oleg

Nov 24, 2009 at 1:10 AM

Oleg,

Yes that's exactly what I am trying to do.

But it seem that every times I compile T4 Toolbox will run. Maybe I am missing something in short I just need a way to tell the Generator to run on demand (for now anyway). (Note I am using the older version that SharpArchitecture use (9.5.20.1). Has T4 Toolbox generation behavior change since then?

Coordinator
Nov 24, 2009 at 11:18 AM

This behavior hasn't changed. T4 template transformation occurs only on demand - either when you save a .tt file or select "Run Custom Tool" from its context menu in solution explorer.

Nov 28, 2009 at 7:44 AM

Oleg,

Thanks for clarifying that. I think what happened was that I commented out the Generator.Run() which resave the file that's what causing it to do the regeneration. I guess I just need to leave it alone next time.

I still think that it would be more intuitive that if generator.Run() is commented out it shouldn't do anything. But it's not a huge deal breaker since I understand the expected behavior now.

Feb 19, 2010 at 4:27 PM
olegsych wrote:

I was reluctant to implement this in the core framework when the patch was initially submitted because I don't think it's a good idea to generate "empty" output files, files that serve as a "starting" point for developers to finish . Having these output files seems convenient when I'm building a code generator. However, once the code generator is ready and I start using it to generate numerous source files, having an empty ouput file per generated class quickly clutters the project. Creating these partial classes by hand is quick enough, so I tend to err on the side of long-term simplicity and don't generate them.

However, this request keeps coming up. If you are interested in implementing it, I can help you extend the core framework of T4 Toolbox to support this feature as a Template.Output option.

I have an explicit counter-example to this good practice, which I would agree with in general. That is that I have a set of database entities that I'm modelling through several layers of abstraction (middle tier, client 1, client 2) and for more complicated reasons there are pieces that I just cannot code generate. However, 90% of the functionality I can code gen and am. However, for every one of these code gen'd files I have to create a partial class, and because we're doing DI I need a partial interface as well. I have about 250 db entities, and if we keep it to the simple 3x increase (in some cases I have 4 diff classes per entity), that's 1500 new files I need to create and add (750 code, 750 interface) which is not an insignificant amount of time or work. 

I think in a case where you know every gen'd file will have a hand crafted counterpart, this feature makes a great deal of sense.

Coordinator
Feb 19, 2010 at 9:12 PM

Point taken, @computerjunky. Let me see if I can squize this into the February release.

Oleg

Coordinator
Mar 14, 2010 at 12:18 AM

This has been implemented in the March build. Set Output.PreserveExistingFile to True. More here: http://www.olegsych.com/2010/03/t4-tutorial-integrating-generated-files-in-visual-studio-projects/.

Oleg