Fix MilCodeGen #6135

ThomasGoulet73 · 2022-02-16T04:37:39Z

Description

This PR aims to fix MilCodeGen which seems to be very outdated and it doesn't seem like it ever worked in this repo (It probably worked in the internal repo for Microsoft).

Since it does not seem like it ever worked, I did these changes just by deducing how it's supposed to work so this PR might be totally wrong.

I noticed a couple of changes in the generated code other than white-space and license header where I don't know whether the source of truth is the new generated code or the existing generated code (It's mostly comments that differ between both).

This PR unblocks PRs like this one: #5651.

Customer Impact

None.

Regression

No.

Testing

I tested this fix locally using this command build -projects "src\Microsoft.DotNet.Wpf\src\WpfGfx\codegen\mcg\mcg.proj" and by looking at the changes and doing a normal build afterwards.

Risk

I think the risk is pretty low since it's only used for development in this repo and the worse that can happen is to be unable to run MilCodeGen (It didn't work before this PR so it can only be an improvement). All the generated code is checked out in Git so it's easier to validate that the generated code is good.

ThomasGoulet73 · 2022-02-16T04:44:57Z

src/Microsoft.DotNet.Wpf/src/WpfGfx/codegen/mcg/tools/InvokeCSP.cmd

@@ -21,17 +21,8 @@ pushd %~dp0\..

 call %~dp0\SetClrPath.cmd

-copy %CspExePath%\csp.exe %clrpath%


I don't know the context for copying csp to clrpath but it seemed unnecessary.

ThomasGoulet73 · 2022-02-16T04:46:39Z

src/Microsoft.DotNet.Wpf/src/WpfGfx/codegen/mcg/tools/SetClrPath.cmd

@@ -1,6 +1 @@
 set clrpath=%MANAGED_TOOLS_ROOT%\%MANAGED_REFS_VERSION%


I didn't know how to get clrpath automatically. It's the path used by sn.exe but I couldn't find where to get it.

ThomasGoulet73 · 2022-02-16T04:48:57Z

src/Microsoft.DotNet.Wpf/src/WpfGfx/codegen/mcg/tools/GenerateElements.cmd

-
-:: Revert any generated files which haven't changed
-if {%_NoRevertFlag%} EQU {} call %~dp0\tf_GeneratedFiles %OutputDir% %~dp0\GeneratedElements.txt revert -a
+call %~dp0\InvokeCSP.cmd %Options% -rsp:main\Elements.rsp -enableCsPrime -r:"System.Xml.dll" -- %_SdFlag% -xmlFile:xml\Elements.xml -dataType:MS.Internal.MilCodeGen.ResourceModel.CG -o:"%OutputDir%"


I removed the argument -clrdir and the full path in the argument -r. It works without it and avoids the need of finding the path for .Net Framework reference assemblies.

ThomasGoulet73 · 2022-02-16T04:50:48Z

src/Microsoft.DotNet.Wpf/src/WpfGfx/codegen/mcg/main/Elements.rsp

@@ -0,0 +1,19 @@
+-s:main\SimpleGenerator.cs


I created this file and Resources.rsp by hand because they didn't exist. They were probably missed when adding the code in this repo because *.rsp files are excluded in the gitignore.

There are 3 .rsp files in internal source, Elements Resources and SimpleGenerator.
I am still looking at these and trying to make sense, but in the meanwhile here are the contents of the 3 files :

Elements.rsp

-rsp:main\SimpleGenerator.rsp -s:Generators\Elements.cs -s:Generators\IAnimatableHelper.cs -s:ResourceModel\Elements.cs -s:ResourceModel\Generated\Elements.cs

Resources.rsp

-s:main\ResourceGenerator.cs -s:main\cmdline\IndentedWriter.cs -s:main\cmdline\Utility.cs -s:main\cmdline\CommandLineArguments.cs -s:Runtime\DelimitedList.cs -s:Runtime\OrList.cs -s:Runtime\CodeSink.cs -s:Runtime\FileCodeSink.cs -s:Runtime\StringCodeSink.cs -s:Runtime\ParameterList.cs -s:Runtime\XmlLoader.cs -s:Runtime\GeneratorMethods.cs -s:ResourceModel\Command.cs -s:ResourceModel\Enum.cs -s:ResourceModel\Field.cs -s:ResourceModel\Primitive.cs -s:ResourceModel\Realization.cs -s:ResourceModel\RenderData.cs -s:ResourceModel\Resource.cs -s:ResourceModel\ResourceModel.cs -s:ResourceModel\Struct.cs -s:ResourceModel\Type.cs -s:ResourceModel\Array.cs -s:helpers\CodeGenHelpers.cs -s:helpers\CollectionHelper.cs -s:helpers\EmptyClassHelper.cs -s:helpers\ManagedStyle.cs -s:helpers\PaddedCommand.cs -s:helpers\StringHelpers.cs -s:helpers\Style.cs -s:helpers\EnumHelper.cs -s:helpers\StructHelper.cs -s:generators\AnimatableTemplate.cs -s:generators\AnimationBaseTemplate.cs -s:generators\AnimationClockResource.cs -s:generators\AnimationResourceTemplate.cs -s:generators\AnimationTemplate.cs -s:generators\AnimationUsingKeyFramesTemplate.cs -s:generators\CommandProcessMessage.cs -s:generators\CommandStructure.cs -s:generators\CommandType.cs -s:generators\DiscreteKeyFrameTemplate.cs -s:generators\DuceResource.cs -s:generators\EasingKeyFrameTemplate.cs -s:generators\FrameworkElementTemplate.cs -s:generators\IAnimatableHelper.cs -s:generators\KeyFrameTemplate.cs -s:generators\KeyFrameCollectionTemplate.cs -s:generators\LinearKeyFrameTemplate.cs -s:generators\ManagedEnum.cs -s:generators\ManagedResource.cs -s:generators\ManagedStruct.cs -s:generators\PolySegmentTemplate.cs -s:generators\RenderData.cs -s:generators\ResourceFactory.cs -s:generators\ResourceType.cs -s:generators\SplineKeyFrameTemplate.cs -s:generators\Template.cs -s:generators\MiscDef.cs -s:generators\MilRenderTypesGenerated.cs -s:generators\WincodecPrivateGenerated.cs

SimpleGenerator.rsp

-s:main\SimpleGenerator.cs -s:main\cmdline\IndentedWriter.cs -s:main\cmdline\Utility.cs -s:main\cmdline\CommandLineArguments.cs -s:Runtime\DelimitedList.cs -s:Runtime\OrList.cs -s:Runtime\CodeSink.cs -s:Runtime\FileCodeSink.cs -s:Runtime\StringCodeSink.cs -s:Runtime\ParameterList.cs -s:Runtime\GeneratorMethods.cs -s:ResourceModel\CG.cs -s:helpers\ManagedStyle.cs -s:helpers\Style.cs

src/Microsoft.DotNet.Wpf/src/WpfGfx/tools/csp/Project.cs

kant2002 · 2022-02-16T06:48:18Z

You stole my next target 😄

ThomasGoulet73 · 2022-02-18T00:09:21Z

I pushed a bunch of commit to try to reduce the diff but I couldn't reduce it much further because the bulk of the changes in generated code come from whitespace added because of conditional code which adds an empty line (Example: This code generates these whitespaces).

omariom · 2022-03-08T03:42:28Z

With 308 changed files it's almost impossible to understand what has changed :/

kant2002

I think would be worth to keep build changes from style formatting changes separately. I have hopes that this allow speedup of the review, or at least schedule it for testing.

kant2002 · 2022-03-09T17:24:16Z

...crosoft.DotNet.Wpf/src/PresentationCore/System/Windows/Generated/TextDecorationCollection.cs

@@ -250,12 +250,15 @@ internal void RemoveAtWithoutFiringPublicEvents(int index)

                WritePreamble();

-                if (!Object.ReferenceEquals(_collection[index], value))
+                if (!Object.ReferenceEquals(_collection[ index ], value))


These kind of changes is against current code style. I assume that it would be hard to convince WPF team on this kind of changes. Also they are reduce ability understand what's going on.

kant2002 · 2022-03-09T17:24:32Z

...crosoft.DotNet.Wpf/src/PresentationCore/System/Windows/Generated/TextDecorationCollection.cs

                    OnFreezablePropertyChanged(oldValue, value);

-                    _collection[index] = value;
+                    _collection[ index ] = value;
+


Do these empty lines needed?

kant2002 · 2022-03-09T17:25:34Z

src/Microsoft.DotNet.Wpf/src/PresentationCore/System/Windows/Media/Generated/DashStyle.cs

@@ -236,8 +245,11 @@ void DUCE.IResource.ReleaseOnChannel(DUCE.Channel channel)

                if (_duceResource.ReleaseOnChannel(channel))
                {
+


Why these two lines? I see these a lot. That's confusing.

ThomasGoulet73 · 2022-05-13T21:53:20Z

@kant2002 @omariom Sorry about the delay, I forgot about this PR. All the files are generated using MilCodeGen without any modifications by hand so all the changes are necessary.

It looks like a lot of generated files were formatted when adding the files to this repo.
For example, TextDecorationCollection from referencesource.microsoft.com is a lot closer to this repo with the changes in this PR than without this PR. TextDecorationCollection in this repo with the changes in this PR vs TextDecorationCollection in this repo without the changes in this PR. TextDecorationCollection did not change since the initial commit of PresentationCore.

ThomasGoulet73 · 2024-05-01T01:29:22Z

I rebased, fixed the conflicts and synced MilCodeGen to the generated code since changes to generated files was merged into main.

@dotnet/wpf-developers: Could this be considered for the next test pass ? MilCodeGen is outdated compared to the generated files in the main branch because generated files get changed manually because we cannot run MilCodeGen.

ThomasGoulet73 · 2025-01-28T04:31:28Z

I rebased to fix the conflicts.

I had to restore a bunch of usings that were removed because they were unused but the problem is that they were removed from files that were generated from the same template and some file didn't need the usings and some did so to keep it unified all files generated from the same template have the same usings. I did remove usings that none of the files generated from the same template needed.

dipeshmsft · 2025-02-05T06:52:56Z

@ThomasGoulet73, I have started looking at this PR. I was going through the README files in this area, and my general understanding is that MilCodeGen uses the csp.exe tool to do the codegen.

It can basically be divided into three parts I guess :

Fixing csp.csproj to get it to build and run.
Fixing milcodegen to get it to build and run.
Fixing templates and regenerating the code files.

If my understanding is correct, will it be possible to break the PR into smaller PRs that do the things above.

Also regarding fixing the csp.exe tool, there are some unit tests for the Csp tool were you able to get them to run ?

I will try to run mcg for .NET Framework codebase, I am guessing there it will be in a running state, as the targets files imported in csp.csproj and other files are present there.

dipeshmsft · 2025-02-05T16:18:32Z

I will try to run mcg for .NET Framework codebase, I am guessing there it will be in a running state, as the targets files imported in csp.csproj and other files are present there.

I tried running MCG of Framework codebase and it works there. I will try to match the steps here with logs that are generated from the run. If possible, I will attach a log file for the run.

ThomasGoulet73 · 2025-02-06T00:03:21Z

@ThomasGoulet73, I have started looking at this PR. I was going through the README files in this area, and my general understanding is that MilCodeGen uses the csp.exe tool to do the codegen.

It can basically be divided into three parts I guess :

Fixing csp.csproj to get it to build and run.

Fixing milcodegen to get it to build and run.

Fixing templates and regenerating the code files.

If my understanding is correct, will it be possible to break the PR into smaller PRs that do the things above.

Yes, @h3xds1nz also suggested spliting it here #10310 (comment).

What I think I'm going to do is:

Open a PR to fix CSP and MilCodeGen, to be able to run the codegen (I don't see the value in splitting those in 2 PR).
Open a PR where I update the CodeGen to align templates with the current generated code to reduce diff.
Open a PR for each project that contains generated code where I only update the generated code.

The PRs 1 and 2 won't include changes to shipping code so it will be easy to review. The PRs in step 3 will include changes to shipping code but it shouldn't change the behavior of the code, and since it will be 1 PR per project it will be easier to review.

Also regarding fixing the csp.exe tool, there are some unit tests for the Csp tool were you able to get them to run ?

I can't seem to find those tests you're talking about, could you link me those tests ?

ThomasGoulet73 · 2025-02-06T02:28:37Z

I can't seem to find those tests you're talking about, could you link me those tests ?

Nevermind, I just found it and I'm working on fixing it. Could you check if you have the file CspProject.rsp in your internal repo, it seems to be missing from this repo (For the same reason as #6135 (comment)).

dipeshmsft · 2025-02-06T03:20:25Z

Could you check if you have the file CspProject.rsp in your internal repo, it seems to be missing from this repo.

@ThomasGoulet73 Here's the content of that file

-main:MS.Internal.Csp.Test.CspProjectTest
-s:CspProjectTest.cs

Open a PR to fix CSP and MilCodeGen, to be able to run the codegen (I don't see the value in splitting those in 2 PR).

Open a PR where I update the CodeGen to align templates with the current generated code to reduce diff.

Open a PR for each project that contains generated code where I only update the generated code.

This sounds like a good division.

ThomasGoulet73 · 2025-02-06T03:27:14Z

Thanks @dipeshmsft, I was able to reverse engineer the content CspProject.rsp when I was working on #10417 but it helps to validate that what I did was correct.

This sounds like a good division.

I opened #10417 as the first part of this work, could you prioritize it ? Realistically it's very low to no risk since it doesn't affect shipping code and worse case scenario we can always revert it.

dipeshmsft · 2025-02-06T03:56:05Z

I opened #10417 as the first part of this work, could you prioritize it ? Realistically it's very low to no risk since it doesn't affect shipping code and worse case scenario we can always revert it.

I am already on it. There is still some part of MCG that I haven't gone through, but I will be mostly done by tomorrow.

h3xds1nz · 2025-02-06T09:11:18Z

@dipeshmsft Great to see progress on this one, thanks :)

dipeshmsft · 2025-02-11T16:08:13Z

@ThomasGoulet73 , I think we can go ahead and close the PR now as most of the other MilCodeGen PRs are already merged in main.

ThomasGoulet73 · 2025-02-12T00:32:48Z

@ThomasGoulet73 , I think we can go ahead and close the PR now as most of the other MilCodeGen PRs are already merged in main.

Yes, thanks a lot for reviewing all of this.

ThomasGoulet73 requested a review from a team as a code owner February 16, 2022 04:37

ghost added the PR metadata: Label to tag PRs, to facilitate with triage label Feb 16, 2022

ghost requested review from dipeshmsft, singhashish-wpf and SamBent February 16, 2022 04:37

ThomasGoulet73 commented Feb 16, 2022

View reviewed changes

kant2002 reviewed Mar 9, 2022

View reviewed changes

ThomasGoulet73 mentioned this pull request Mar 17, 2022

Run some built-in .NET SDK perf analyzers over several WPF assemblies #6274

Merged

ThomasGoulet73 mentioned this pull request May 9, 2022

Remove FriendAccessAllowedAttribute usage #6561

Merged

dipeshmsft self-assigned this Jun 18, 2022

ghost assigned ThomasGoulet73 Jun 18, 2022

pchaurasia14 added the Community Contribution A label for all community Contributions label Jul 20, 2022

ThomasGoulet73 mentioned this pull request Feb 9, 2023

Replace resource constants with properties in PresentationCore #7519

Merged

ThomasGoulet73 force-pushed the fix-milcodegen-build branch from 3b939e3 to 4fdf462 Compare May 1, 2024 01:21

ThomasGoulet73 mentioned this pull request Dec 16, 2024

Add readonly to members in Rect #10156

Open

ThomasGoulet73 mentioned this pull request Jan 26, 2025

Disable code analysis for generated code #10337

Merged

ThomasGoulet73 added 9 commits January 27, 2025 23:22

Fix milcodegen build

63023db

Fix MilCodeGen

692498b

Fix whitespace

bcf8c7b

Fix C# license generation

6455d01

Fix AnimationTemplate whitespace

0fb6591

Fix FrameworkElementTemplate.cs

a57bdf7

Sync MilCodeGen to generated code

1667914

Sync MilCodeGen to generated code

0f144e8

Run MilCodeGen

db55009

ThomasGoulet73 force-pushed the fix-milcodegen-build branch from 4fdf462 to db55009 Compare January 28, 2025 04:28

ThomasGoulet73 requested a review from a team as a code owner January 28, 2025 04:28

ThomasGoulet73 mentioned this pull request Feb 6, 2025

Fix MilCodeGen and Csp build #10417

Merged

ThomasGoulet73 closed this Feb 12, 2025

github-actions bot locked and limited conversation to collaborators Mar 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix MilCodeGen #6135

Fix MilCodeGen #6135

ThomasGoulet73 commented Feb 16, 2022 •

edited

Loading

ThomasGoulet73 Feb 16, 2022

ThomasGoulet73 Feb 16, 2022

ThomasGoulet73 Feb 16, 2022

ThomasGoulet73 Feb 16, 2022

dipeshmsft Feb 5, 2025

kant2002 commented Feb 16, 2022

ThomasGoulet73 commented Feb 18, 2022

omariom commented Mar 8, 2022

kant2002 left a comment

kant2002 Mar 9, 2022

kant2002 Mar 9, 2022

kant2002 Mar 9, 2022

ThomasGoulet73 commented May 13, 2022

ThomasGoulet73 commented May 1, 2024 •

edited

Loading

ThomasGoulet73 commented Jan 28, 2025 •

edited

Loading

dipeshmsft commented Feb 5, 2025

dipeshmsft commented Feb 5, 2025

ThomasGoulet73 commented Feb 6, 2025

ThomasGoulet73 commented Feb 6, 2025

dipeshmsft commented Feb 6, 2025 •

edited

Loading

ThomasGoulet73 commented Feb 6, 2025

dipeshmsft commented Feb 6, 2025

h3xds1nz commented Feb 6, 2025

dipeshmsft commented Feb 11, 2025

ThomasGoulet73 commented Feb 12, 2025

		@@ -21,17 +21,8 @@ pushd %~dp0\..

		call %~dp0\SetClrPath.cmd

		copy %CspExePath%\csp.exe %clrpath%

		@@ -1,6 +1 @@
		set clrpath=%MANAGED_TOOLS_ROOT%\%MANAGED_REFS_VERSION%

		@@ -236,8 +245,11 @@ void DUCE.IResource.ReleaseOnChannel(DUCE.Channel channel)

		if (_duceResource.ReleaseOnChannel(channel))
		{

Fix MilCodeGen #6135

Fix MilCodeGen #6135

Conversation

ThomasGoulet73 commented Feb 16, 2022 • edited Loading

Description

Customer Impact

Regression

Testing

Risk

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kant2002 commented Feb 16, 2022

ThomasGoulet73 commented Feb 18, 2022

omariom commented Mar 8, 2022

kant2002 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThomasGoulet73 commented May 13, 2022

ThomasGoulet73 commented May 1, 2024 • edited Loading

ThomasGoulet73 commented Jan 28, 2025 • edited Loading

dipeshmsft commented Feb 5, 2025

dipeshmsft commented Feb 5, 2025

ThomasGoulet73 commented Feb 6, 2025

ThomasGoulet73 commented Feb 6, 2025

dipeshmsft commented Feb 6, 2025 • edited Loading

ThomasGoulet73 commented Feb 6, 2025

dipeshmsft commented Feb 6, 2025

h3xds1nz commented Feb 6, 2025

dipeshmsft commented Feb 11, 2025

ThomasGoulet73 commented Feb 12, 2025

ThomasGoulet73 commented Feb 16, 2022 •

edited

Loading

ThomasGoulet73 commented May 1, 2024 •

edited

Loading

ThomasGoulet73 commented Jan 28, 2025 •

edited

Loading

dipeshmsft commented Feb 6, 2025 •

edited

Loading