Codec string to byte conversion utilities?

[njlr]

I like to hand-roll serialization so that I have full control, but this gives me an IEventCodec<'Event, string, 'Context>.
My category expects an IEventCodec<'Event, ReadOnlyMemory<byte>, 'Context> so a conversion is required.

I came up with these utilities, but perhaps they are already provided?

open FsCodec
open FsCodec.Core

type Encoder with
  static member Unicode(native : IEventCodec<'Event, string, 'Context>) : IEventCodec<'Event, ReadOnlyMemory<byte>, 'Context> =
    EventCodec.mapBodies
      (fun (x : string) -> Encoding.Unicode.GetBytes(x) |> ReadOnlyMemory<byte>)
      (fun (x : ReadOnlyMemory<byte>) -> Encoding.Unicode.GetString(x.Span))
      native

  static member UTF8(native : IEventCodec<'Event, string, 'Context>) : IEventCodec<'Event, ReadOnlyMemory<byte>, 'Context> =
    EventCodec.mapBodies
      (fun (x : string) -> Encoding.UTF8.GetBytes(x) |> ReadOnlyMemory<byte>)
      (fun (x : ReadOnlyMemory<byte>) -> Encoding.UTF8.GetString(x.Span))
      native

If desired can also send a PR.

Thanks!

[bartelink]

For helpers, I'd follow the general pattern in https://github.com/jet/FsCodec/blob/master/src/FsCodec.SystemTextJson/Interop.fs:

• provide atomic helpers, i.e. static member StringToUtf8(x : string) -> Encoding.UTF8.GetBytes(x) |> ReadOnlyMemory<byte>
• alongside that, provide an extension method that uses those, i.e. ToUtf8Codec

---

The next question is whether FsCodec should have support for string Codecs in the box

1. in general, going via strings involves lots of allocations, with no ability to do pooling etc (vs ROM with STJ etc)
2. for STJ, JsonElement is the name of the game
3. For others, ROM or byte[] is native
4. (so for any real situation, conversion from / having an extra hop via string is messy and extra work)

If you poke around the API surfaces of STJ etc, you pretty quickly work out that strings should be a last resort. In the context of FsCodec, the general idea is that you can avoid thinking about such details.

So in terms of the best way of truly addressing the issue, I'd be looking for a way to put the conversion/interop on the Thoth.Json side (i.e. put interop helpers there and/or request a way to codec to/to ROM directly on that side).

---

If that doesnt convince you it's a bad idea to put it in the box, then I'd insist on the PR being presented as:

• cordoned off in a FsCodec.StringInterop namespace (no default autocomplete where people end up with random ToUtf8Codec/ToUnicodeCodec hops without thinking it through
• with a heading comment that conveys the fact that this is a stopgap solution and why
• some of the tests that do utf8/utf16 conversions could probably use the helpers

Thoughts?

[njlr]

Everything you have put makes sense, and I wonder if a more efficient integration with Thoth is possible.

The particular Thoth back-end I have in this case is Newtonsoft.Json.

Allow me to explain the scenario...

Suppose an Event type like this 🏁

type RaceStarted =
  {
    StartedAt : DateTime
    Racers : string list
  }

type RacerFinished =
  {
    FinishedAt : DateTime
    Racer : string
  }

type Event =
  | RaceStarted of RaceStarted
  | RacerFinished of RacerFinished

We can define (or generate) Thoth encoders / decoders:

module RaceStarted =
  
  let encode : Encoder<RaceStarted> =
    fun (x : RaceStarted) ->
      Encode.object 
        [
           "startedAt", Encode.datetime x.StartedAt
           "racers", Encode.list (List.map Encode.string x.Racers)
        ]

  let decode : Decoder<RaceStarted> =
    Decode.object 
      (fun get ->
        {
          StartedAt = get.Required.Field "startedAt" Decode.datetime
          Racers = get.Required.Field "racers" (Decode.list Decode.string)
        })

// etc...

So now we have functions:

module RaceStarted =

  val encode : RaceStarted -> Newtonsoft.Json.Linq.JToken

  val decode : string -> Newtonsoft.Json.Linq.JToken -> Result<RaceStarted, DecoderError>


module RacerFinished =

  val encode : RacerFinished -> Newtonsoft.Json.Linq.JToken

  val decode : string -> Newtonsoft.Json.Linq.JToken -> Result<RacerFinished, DecoderError>

As you can see, these operate on JToken rather than string.

Note the first string given to decode is just the JSON-path (so "$"), not the JSON to be decoded.

I just can't figure out where in the FsCodec API to best integrate these.

[bartelink]

Hm, I've never been down the road of wanting to manually encode/decode things for event bodies.

My basic stance is that in an event sourced system:

• the JSON will outlive the code, so the KISS principle is key. Any custom codec logic is asking for trouble.
• your EventType is a key tool for versioning. If you vary the representation and/or add conditions or upconversions
• make it Event2
• but really just use standard upconversion conventions
• add a new field as an option
• let evolve deal with the None or the upconversion from Event to Event2
• (lots of docs in the readme about this scheme/philosophy)

Next thing: Newtonsoft is dead. Really dead. Its owner hates it. Yes its not going anywhere. But also, basic bugfixes don't happen and don't get merged if they do.

And... STJ is simply the winner. And it can encode to JsonElement, which when squinted at is similar to JToken. The Interop in FsCodec.Stj bridges between that and ROM or byte[]. (for CosmosStore, it consumes JsonElement directly, and that would make sense for any JSON document database)

So I'd strongly suggest using FsCodec.SystemTextJson and calling it a day. While I can see how Thoth can make sense as an explict parsing and rendering mechanism where you are exposing a variety of versioned interfaces to a variety of backends and evolving that over time, encoding event bodies is simply a different game, and all the things that make Thoth good for other contexts are counter-productive and/or lead you down inappropriate paths in the event sourced event body roundtripping context

---

If you choose to ignore the above, I believe you should be able to map ROM event bodies to JToken via .Parse or similar, and presumably do the inverse to UTF8 encode a JToken encoded version? The pattern for most of that route should be in how the STJ Interop.fs works (i.e. make a .ToJtokenCodec converter that takes UTF-8 blobs and converts them to JToken)

[njlr]

Hm, I've never been down the road of wanting to manually encode/decode things for event bodies.

My basic stance is that in an event sourced system:

• the JSON will outlive the code, so the KISS principle is key. Any custom codec logic is asking for trouble.

Agree, which is why I want to decouple the domain type definitions from the JSON encoding / decoding.
The Thoth code can be generated from a JSON-schema, for example.

• your EventType is a key tool for versioning. If you vary the representation and/or add conditions or upconversions

• make it Event2

• but really just use standard upconversion conventions

• add a new field as an option

• let evolve deal with the None or the upconversion from Event to Event2

• (lots of docs in the readme about this scheme/philosophy)

This makes sense, although a portable definition outside of the code (e.g. a growing versioned list of JSON-schemas) can achieve this?

Next thing: Newtonsoft is dead. Really dead. Its owner hates it. Yes its not going anywhere. But also, basic bugfixes don't happen and don't get merged if they do.

And... STJ is simply the winner. And it can encode to JsonElement, which when squinted at is similar to JToken. The Interop in FsCodec.Stj bridges between that and ROM or byte[]. (for CosmosStore, it consumes JsonElement directly, and that would make sense for any JSON document database)

Yes I am a bit behind and need to move to STJ. Later Thoth versions use STJ.

So I'd strongly suggest using FsCodec.SystemTextJson and calling it a day. While I can see how Thoth can make sense as an explict parsing and rendering mechanism where you are exposing a variety of versioned interfaces to a variety of backends and evolving that over time, encoding event bodies is simply a different game, and all the things that make Thoth good for other contexts are counter-productive and/or lead you down inappropriate paths in the event sourced event body roundtripping context

If you choose to ignore the above, I believe you should be able to map ROM event bodies to JToken via .Parse or similar, and presumably do the inverse to UTF8 encode a JToken encoded version? The pattern for most of that route should be in how the STJ Interop.fs works (i.e. make a .ToJtokenCodec converter that takes UTF-8 blobs and converts them to JToken)

Will take a look and reconsider. Thanks! 👍

[bartelink]

JSON schemas and codegen can work. And some might see the schema artifacts as a win and prefer to work at that level.

protobufs can work and are way more efficient, but you would not do it without tooling. If you are using that somewhere in the building, you are getting perf if the cost or learning/fighting/troubleshooting the tooling is effectively zero for you and al future maintainers.

messagepack can work.

---

So, it comes down to what your Enterprise demands and can easily consume/support.

For most Enterprises that don't have a stance/requirement and are using F#, I strongly recommend using FsCodec/TypeShape UnionEncoder based on:

• no code, event bodies can be DUs or anonymous DUs
  • no tests required to make sure no typos in your literal strings in the codecs
  • way less to compile and traverse for you in your IDE or future maintainers LLMs
• no debugging/wondering ever about how the JSON for a given structure will look
• Good Enough perf (it's not going to beat protobuf, but most stuff is going to be IO bound, and Equinox's caching also strips out significant codec activity compared to a what you might spin up by default)
• Much harder to end up with bad patterns and encodings in your events. Records, options and arrays FTW
• Boring aggregates can follow the pattern without being dragged into a boilerplate battle or having to argue why you want to special case those

Ultimately, because things are pluggable, you can have 20 Aggregates with default codec behavior, and you have dropping a level to manual coding encoding in your toolbox to apply when you need. And then people can examine the code and see why this one is special, vs perpetually traversing 20 sets of boilerplate and wondering how many of them actually have bespoke logic - is it 1, 5 or 20?

---

Bottom line/restating my point - roundtripping bodies in event sourcing is and should remain a very different exercise to rendering/parsing/versioning of payloads for external systems

Multiple reads of https://eiriktsarpalis.wordpress.com/2018/10/30/a-contract-pattern-for-schemaless-datastores are recommended ;)

---

NOTE if you are using DynamoStore or CosmosStore, view models can follow the same pattern

• use the same auto codec/decode scheme with no busywork (just use AcessStrategy.RollingState
• reads/writes/caching/logging/troubleshooting work identically to how it does for event sourced info, but you just have a single document/item being compared and swapped under control of the Decider.

[bartelink]

Closing for now; If you arrive at a distilled helper that you're happier with in a prod context, we should revisit and consider adding. For now I'm saying the in-the-box state has worked for others and does not IMO send people down paths they don't need to be sent down