External Mapping Storage

[PG-RichT]

Hi there,

Are there any plans to support an external source for the Mappings data?

I ask because I've started an exploratory feature branch to support Cosmos DB and a colleague suggested getting in touch in case this is on your roadmap? If not, do you have any suggestions or requirements that could ease the merging of this code into your master repo at some point?

Here's my very quick stab at how it could be done: https://github.com/WireMock-Net/WireMock.Net/compare/master...PG-RichT:WireMock.Net:cosmos_backend?expand=1

Thanks,
Rich

[StefH]

So instead of storing the Mappings in memory, you want to store the mappings in CosmosDB?

[PG-RichT]

Exactly :) We deploy a Wiremock server for a large suite of integration tests so it would great to be able to scale up the server instances as we dynamically add scenarios for specific tests.

[StefH]

In that case, some ideas for the approach...

1. the IMappingProvider interface can be added to the WireMock.Net.Abstractions project
2. the LegacyMappingProvider can be renamed to DefaultMappingProvider and added to WireMock.Net project
3. a new project WireMock.Net.MappingProviders.Cosmos should be created
4. the correct implementation for IMappingProvider should be used if the configuration defines Cosmos and if the WireMock.Net.MappingProviders.Cosmos is included. The logic for this can be using the updated PluginLoader as defined in Extend TypeLoader #1069

[PG-RichT]

Awesome, thanks for the pointers. Will keep you posted.

[PG-RichT]

Can't move IMappingProvider to abstractions. It has a dependency on IMapping which itself has a dependency on a whole bunch of stuff in the Wiremock.net project.

[StefH]

Good point.

In that case keep it in the main project.

[PG-RichT]

Hmmm looking more and more like this won't be suitable for everyone; The Azure Cosmos SDK doesn't support 4.5.2 so I'll need to drop support for that.

[StefH]

That is not issue if WireMock.Net.MappingProviders.Cosmos is a separate project which does support the frameworks which are supported by Cosmos SDK.

The same is for GraphQL and gRPC, that is only available for the newer frameworks.

[PG-RichT]

Unfortunately due to the dependencies of IMapping, we can't have an external project reference the WireMock.net project for the Interface and then the Wiremock.net project reference a Cosmos project. It causes a circular dependency.

Implementing this is very low level, not really plugin. I think this is a fork forever...

[StefH]

It is possible, WireMock.Net.Matchers.CSharpCode uses the same idea.

The interface ICSharpCodeMatcher is defined in main project and the implementation CSharpCodeMatcher is defined in WireMock.Net.Matchers.CSharpCode.

Loading the type is done here:

WireMock.Net/src/WireMock.Net/Serialization/MatcherMapper.cs

Line 56 in 2364866

return PluginLoader.Load<ICSharpCodeMatcher>(matchBehaviour, matchOperator, stringPatterns);

(Maybe PluginLoader is a wrong name, maybe better like TypeLoader or something...)

[PG-RichT]

@StefH Ahhhhh ok, I'm starting to understand how you've done it now. Okay, I'll keep persevering 👍🏼

[StefH]

@StefH Ahhhhh ok, I'm starting to understand how you've done it now. Okay, I'll keep persevering 👍🏼

Ok. Good to hear. In case you would like to build this logic and need an updated WireMock.Net, please tell me. Then I'll merge .

Also think on the naming:
Currently it's PluginLoader.Load and PluginLoader.LoadByFullName, and this last is the one you probably need.

So maybe:

• TypeActivator.Activate
• TypeActivator.ActivateByFullName

or

• TypeLoader.Activate
• TypeLoader.ActivateByFullName

Are better names?

[PG-RichT]

Your new code will be essential so please go ahead. With regards to the name, TypeLoader has clearer intentions.

Thank you for your help with this, it is greatly appreciated.

[StefH]

@PG-RichT
Code is merged.

[StefH]

Would you like to create a PR for your new code? Then I can already add comments on the code while you develop?

[PG-RichT]

@StefH Will do as soon I've resolved my current blocker - serializing the Interfaces used throughout a Mapping object in order to store them in the database.

[StefH]

Yes, serializing the IMapping / Mapping to JSON is not easy.

What you could try is use the MappingConverter to convert a IMapping <> MappingModel (this is the same is used on the HTTP admin interface).

However, you will lose some functionality like the Func<> callbacks.

[StefH]

@PG-RichT
I did some thinking.... 🤔

What you actually can do is just implement the IFileSystemHandler interface.
https://github.com/WireMock-Net/WireMock.Net/blob/master/src/WireMock.Net.Abstractions/Handlers/IFileSystemHandler.cs

This interface is used to read and write all the files needed by WireMock.Net , this also includes the mapping (MappingModel -JSON) files.

[PG-RichT]

@StefH Thanks for that. I have started an approach (I don't know if it will work yet) that recognises an internal mapping i.e. one used by wiremock only, versus a user mapping.

Internal mappings will be stored in a concurrent dictionary like before, user mappings in cosmos.

I don't know how this will affect performance or if it will even work yet but let's see...

[StefH]

There is technically no difference between a wiremock internal or a user mapping. These are all Mappings. So I think the best approach is : saving/reading the MappingModels as JSON in the CosmosDB using the IFileSystemHandler .

[PG-RichT]

@StefH Apologies for a delay in my response. I have been assessing the implementation using the IFileSystemHandler and unfortunately I don't think it meets the requirements I would want out of backend storage.

Ideally the database of mappings would be called per request so any number of Wiremock instances could be spun up. If we go down the IFileSystemHandler route we introduce potential latency as each instance responds to an event telling it that 'the database has been updated, go refresh your in-memory mappings'. That latency might be tiny but the fact that it exists is enough to put me off.

Instead I have been looking at other infrastructure approaches to achieve our end-goal, sticky sessions appears to be simplest.

I wanted to thank you for your help so I could come to this conclusion though and for all the hard work and effort you put into maintaining this repo. Thank you.