[tools] TypeMap dumper utility by grendello · Pull Request #5541 · dotnet/android

grendello · 2021-01-25T15:03:41Z

Utility to read and dump to files/screen the typemap contents found in
libxamarin-android.so

radekdoulik

LGTM

It would be nice if it can later provide a library (in form of nuget) to read type maps, so that it can be used as PackageReference in other tools. Here I have apkdiff in mind, where it can show changes in the typemaps.

radekdoulik · 2021-01-25T17:54:57Z

There's this error on CI:

NU1605: Detected package downgrade: K4os.Compression.LZ4 from 1.2.6 to 1.1.11. Reference the package directly from the project to select a different version.

It looks like the CI just didn't pickup last commit?

radekdoulik · 2021-01-25T17:55:03Z

/azp run

azure-pipelines · 2021-01-25T17:55:40Z

Azure Pipelines successfully started running 1 pipeline(s).

jonathanpeppers · 2021-01-25T20:05:59Z

+    <OutputPath>../../bin/$(Configuration)/bin/tmt</OutputPath>
+    <OutputType>Exe</OutputType>
+    <LibZipSharpBundleAllNativeLibraries>true</LibZipSharpBundleAllNativeLibraries>
+    <LangVersion>$(LangVersion)</LangVersion>


I think you can leave this line out if you like.

I keep copying it over and over... :)

dellis1972 · 2021-01-25T20:20:23Z

There's this error on CI:
NU1605: Detected package downgrade: K4os.Compression.LZ4 from 1.2.6 to 1.1.11. Reference the package directly from the project to select a different version.
It looks like the CI just didn't pickup last commit?

Nope this is because monodroid is referencing the older package. There is a PR to upgrade monodroid that will need to be included in this one.

grendello · 2021-01-25T20:45:18Z

LGTM

It would be nice if it can later provide a library (in form of nuget) to read type maps, so that it can be used as PackageReference in other tools. Here I have apkdiff in mind, where it can show changes in the typemaps.

That's a good idea :) I will work on it at some point - we would probably need to move the reading code to a separate repository.

dellis1972 · 2021-01-26T10:07:56Z

@@ -1,2 +1,2 @@
-xamarin/monodroid:master@ad19471b8478f9f8028c9a3517ad3ca7bca5fb4a
+xamarin/monodroid:master@390c048caaddfc6549ef94d6a80f71e5aa9416bc


can you change this to this commit d9fcaa019c5e2fb14384d47040bbe7f0b8e7695c? its the head of master. It should also fix a few failing tests.

grendello · 2021-01-28T10:48:54Z

Working on some improvements and support for FastDev typemap files

jonpryor · 2021-02-03T19:45:59Z

+`X86_64`
+
+# Command line options
+


This should use a triple-backtick "block" so that it renders as intended.

Please provide an example invocation, e.g. on samples/HelloWorld output.

jonpryor · 2021-02-05T01:09:45Z

+# bin/Debug/bin/tmt/tmt samples/HelloWorld/obj
+# bin/Debug/bin/tmt/tmt samples/HelloWorld/obj/Debug
+#
+$ bin/Debug/bin/tmt/tmt samples/HelloWorld/


I built this branch, and the bin/Debug/bin/tmt/tmt script does not exist.

In order to run the app, I need to use dotnet bin/Debug/bin/tmt/tmt.dll, which is fine and works (as far as it goes), but doesn't match this doc.

Additionally, tools/tmt/run.sh doesn't look like it's supposed to be copied into bin/Debug/bin/tmt/tmt, as that contains an msbuild /r invocation…

Is something missing from this PR which is still on your local tree?

tmt is a netcore app and it produces bin/Debug/tmt/tmt as a native executable, it's not a script:

$ ls -l bin/Debug/bin/tmt/tmt -rwxr-xr-x 1 grendel grendel 86584 Feb 3 22:20 bin/Debug/bin/tmt/tmt $ file bin/Debug/bin/tmt/tmt bin/Debug/bin/tmt/tmt: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=22531e6f94df2fbec39dd3d1ef003bcdcc0288df, stripped

Not sure why you don't have it in your tree.

run.sh will be removed, it's merely a shortcut I use for testing to avoid typing the same commands all over again.

jonpryor · 2021-02-05T01:27:30Z

+  -v, --verbose              Show debug messages
+```
+
+## Sample invocations


Part of the request for sample invocations was to see what output was generated, which also isn't described anywhere. (This is why I built the branch, on the hopes I could see what it was like and consider updating the commit message/README.md… then found that the tmt script didn't exist.)

Now that I have the output, I see that it's a "tab-delimited format":

Java Type Name Managed Type Name Generic? Duplicate? android/accessibilityservice/AccessibilityButtonController Android.AccessibilityServices.AccessibilityButtonController, Mono.Android

This is good, as far as it goes, but there is no good description of what the "Generic?" column is for, and it doesn't appear to be used in the output for samples/HelloWorld.

The Duplicate? column is presumably for when the same Java type is mapped to multiple C# types, a'la 66f7206, but even then I'm not quite sure how to interpret it?

android/accessibilityservice/AccessibilityGestureEvent Android.AccessibilityServices.AccessibilityGestureEvent, Mono.Android android/accessibilityservice/AccessibilityService Android.AccessibilityServices.AccessibilityService, Mono.Android aliased

android/accessibilityservice/AccessibilityService has an aliased entry, but what's the non-aliased entry? Or does aliased mean that there is more than one C# type, period, and indicates nothing about preferred mapping?

Looking at the managed-to-Java mapping, this "no preference" interpretation appears correct, as only one entry is listed Java-to-managed (above):

Managed Type Name Java Type Name Generic? Duplicate? Android.AccessibilityServices.AccessibilityService, Mono.Android android/accessibilityservice/AccessibilityService … Android.AccessibilityServices.AccessibilityServiceInvoker, Mono.Android android/accessibilityservice/AccessibilityService

TL;DR: these files are also part of the usage pattern of this app. They should be likewise described. I'm happy to help with that description, but I can't easily fix the "missing tmt script" problem.

Oh, and of course the Release format differs, so that'll need an explanation as well:

Java Type Name Managed Type Name MVID Token ID … java/lang/Object Java.Lang.Object, Mono.Android dc97aced-a52d-4110-bb03-b57c6368c440 33554768 (0x02000150)

I wonder if these headers should use - instead of space within the description, to make it easier to visually distinguish the tabs, e.g.

Java-Type-Name Managed-Type-Name MVID Token-ID

Part of the request for sample invocations was to see what output was generated, which also isn't described anywhere. (This is why I built the branch, on the hopes I could see what it was like and consider updating the commit message/README.md… then found that the tmt script didn't exist.)

Not sure why your dotnet doesn't produce the tmt binary, I would assume some sort of consistency when building the same project with the same tool...

Java Type Name Managed Type Name Generic? Duplicate? android/accessibilityservice/AccessibilityButtonController Android.AccessibilityServices.AccessibilityButtonController, Mono.Android

This is good, as far as it goes, but there is no good description of what the "Generic?" column is for, and it doesn't appear to be used in the output for samples/HelloWorld.

Where would I put the description? The column headers should be short and I think that, considering we're looking at the listing of C# and Java types, the word would be clear to mean a Generic type. As for the column being empty in samples/HelloWorld output... well, can't do much about it, can I? :)

The Duplicate? column is presumably for when the same Java type is mapped to multiple C# types, a'la 66f7206, but even then I'm not quite sure how to interpret it?

There's not much to interpret here - it's what you presumed originally. Again, the assumption is that the person using the tool knows what typemaps are and how to interpret the word in the context.

android/accessibilityservice/AccessibilityGestureEvent Android.AccessibilityServices.AccessibilityGestureEvent, Mono.Android android/accessibilityservice/AccessibilityService Android.AccessibilityServices.AccessibilityService, Mono.Android aliased

android/accessibilityservice/AccessibilityService has an aliased entry, but what's the non-aliased entry? Or does aliased mean that there is more than one C# type, period, and indicates nothing about preferred mapping?

Jon, what are we trying to achieve here? Do you want the output of the tool to explain how typemaps work and what they are? I hardly think this is the correct place to do it. The format and the workings are documented elsewhere and I really think that the utility is not for general use by every developer using Xamarin.Android, but for forensic examination by those who need to dive deep enough to have to learn what typemaps are (i.e. mostly the XA team) - and if somebody goes that deep to the details of what we put in the APK, I'm sure they will know what "generic", "duplicate" and "aliased" are. If you don't like these terms, please tell me what to put there as I think they're clear enough. Incidentally, the "logical" choice of word for the Duplicate? column would be yes (and no, if we want to raise signal to noise ratio) but with the first line scrolled out of view, a yes (or a no) would be totally confusing. The word aliased hits much closer to target, but if you can think of a word that's more descriptive than that, let's use it (I'd really love to avoid putting a whole sentence in that field)

Looking at the managed-to-Java mapping, this "no preference" interpretation appears correct, as only one entry is listed Java-to-managed (above):

Managed Type Name Java Type Name Generic? Duplicate? Android.AccessibilityServices.AccessibilityService, Mono.Android android/accessibilityservice/AccessibilityService … Android.AccessibilityServices.AccessibilityServiceInvoker, Mono.Android android/accessibilityservice/AccessibilityService

TL;DR: these files are also part of the usage pattern of this app. They should be likewise described. I'm happy to help with that description, but I can't easily fix the "missing tmt script" problem.

I can't fix that either as I don't know why dotnet doesn't produce the binary on your system but creates it on mine. I will instead provide an actual script which will work in both cases.

Do you want the output of the tool to explain how typemaps work and what they are?

No, not the output of the tool; The README.md of the tool. Which is why this comment thread started on the README.md.

And again, I'd be happy to write up that description, I just wasn't sure what some of the values were supposed to mean.

For example, my initial Hello World build did no linking, and thus was huge (yay?), with 6062 Java-to-managed entries and 5376 Managed-to-Java entries.

This should be ~everything in Mono.Android.dll, right? (Perhaps not? There are 10848 TypeDefinitions within Mono.Android.dll, though not every typedef is a distinct binding, so "half-ish" might be a sensible value.)

So why don't any of these entries appear to produce output in the Is-Generic-Type? column? Or am I missing something? I thought we needed the generic column for e.g. Android.Runtime.JavaList<T>?

…which oddly isn't present?!

% grep JavaList x/typemap-v1-Debug-X86_64-managed.txt Android.Runtime.JavaList, Mono.Android java/util/ArrayList duplicate entry

There is a generic type present, but it doesn't have anything in the Is-Generic-Type? column:

% grep JavaArray x/typemap-v1-Debug-X86_64-managed.txt Android.Runtime.JavaArray`1, Mono.Android mono/android/runtime/JavaArray

Why doesn't JavaArray<T> have a value for Is-Generic-Type?

For that matter, why is JavaArray<T> the only generic type in here?

% grep '`' x/typemap-v1-Debug-X86_64-managed.txt Android.Runtime.JavaArray`1, Mono.Android mono/android/runtime/JavaArray

At this point I start wondering if something is wrong with my setup…?

jonpryor · 2021-02-05T14:13:16Z

+				}
+
+				Log.Error ($"Module {m.assembly_name} ({m.module_uuid}) at index {moduleIndex} doesn't contain an entry for managed type with token ID {tokenID}");
+				return (false, null, false, false);


…and this is presumably why I don't see any "generic" types (perhaps): isGeneric is always false.

Though I think this is only for the Release typemap, not the debug typemap, so…? ¯_(ツ)_/¯

Utility to read and dump to files/screen the typemap contents found in `libxamarin-android.so`

grendello requested a review from jonpryor January 25, 2021 15:03

grendello requested review from dellis1972, jonathanpeppers and radekdoulik as code owners January 25, 2021 15:03

grendello force-pushed the typemap-tool branch from dbd4cc1 to 471ef18 Compare January 25, 2021 15:07

dellis1972 reviewed Jan 25, 2021

View reviewed changes

Comment thread tools/tmt/Loader.cs Outdated

radekdoulik approved these changes Jan 25, 2021

View reviewed changes

jonathanpeppers reviewed Jan 25, 2021

View reviewed changes

grendello force-pushed the typemap-tool branch from 471ef18 to ffcc0c9 Compare January 25, 2021 20:43

grendello force-pushed the typemap-tool branch 2 times, most recently from a2c885a to 6770b39 Compare January 26, 2021 09:33

dellis1972 reviewed Jan 26, 2021

View reviewed changes

grendello force-pushed the typemap-tool branch 2 times, most recently from 38ea97f to 538c06d Compare January 26, 2021 11:26

grendello marked this pull request as draft January 28, 2021 10:48

grendello force-pushed the typemap-tool branch 2 times, most recently from 88caa73 to 81f7dc9 Compare January 28, 2021 20:59

jonpryor mentioned this pull request Jan 28, 2021

[tools] Add a tool to decompress XA-compressed assemblies #5536

Merged

grendello marked this pull request as ready for review January 29, 2021 08:14

grendello force-pushed the typemap-tool branch from 81f7dc9 to 21fba85 Compare February 2, 2021 11:38

jonpryor reviewed Feb 3, 2021

View reviewed changes

grendello force-pushed the typemap-tool branch from 21fba85 to 0458ba3 Compare February 3, 2021 21:34

dellis1972 approved these changes Feb 4, 2021

View reviewed changes

jonathanpeppers approved these changes Feb 4, 2021

View reviewed changes

radekdoulik mentioned this pull request Feb 4, 2021

Remove fallback code in GetJavaToManagedType #5593

Closed

jonpryor reviewed Feb 5, 2021

View reviewed changes

grendello force-pushed the typemap-tool branch 2 times, most recently from 30365f5 to e2fd31c Compare February 5, 2021 09:17

jonpryor reviewed Feb 5, 2021

View reviewed changes

grendello force-pushed the typemap-tool branch 7 times, most recently from d21b0f5 to be35ad5 Compare February 9, 2021 18:43

[tools] TypeMap dumper utility

d57dfff

Utility to read and dump to files/screen the typemap contents found in `libxamarin-android.so`

grendello force-pushed the typemap-tool branch from be35ad5 to d57dfff Compare February 9, 2021 21:04

jonpryor merged commit 46f10fe into dotnet:master Feb 10, 2021

grendello deleted the typemap-tool branch February 10, 2021 20:29

github-actions Bot locked and limited conversation to collaborators Jan 24, 2024

		@@ -1,2 +1,2 @@
		xamarin/monodroid:master@ad19471b8478f9f8028c9a3517ad3ca7bca5fb4a
		xamarin/monodroid:master@390c048caaddfc6549ef94d6a80f71e5aa9416bc

Uh oh!

Conversation

grendello commented Jan 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

radekdoulik left a comment

Choose a reason for hiding this comment

Uh oh!

radekdoulik commented Jan 25, 2021

Uh oh!

radekdoulik commented Jan 25, 2021

Uh oh!

azure-pipelines Bot commented Jan 25, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dellis1972 commented Jan 25, 2021

Uh oh!

grendello commented Jan 25, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grendello commented Jan 28, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grendello Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

grendello commented Jan 25, 2021 •

edited

Loading

grendello Feb 5, 2021 •

edited

Loading