Process Queue one by one

[mozesa]

Hello,

I have a workflow in which I feed a queue with certain data, and there is a forked task which processes those data.

...
from cancel in fork(tighteningData | SaveToFile | ListFiles | SaveToDb | DelFiles)
...

where var tighteningData = Queue<RT, string>();.

I noticed that multiple sequences of SaveToFile | ListFiles | SaveToDb | DelFiles are possible to be fired, and unfortunately it is problematic for instance pipes like ListFiles as it can cause to happen, that multiple tasks: list then upload the same file.

What should be the concise way to lower the parallelism to 1 in order that I could process the tightening data (with steps | SaveToFile | ListFiles | SaveToDb | DelFiles one by one).

Thanks for your help in advance.

[mozesa]

Producer.merge solved the problem.

May I know what is the difference between the two?
fork(tighteningData | SaveToFile | ListFiles | SaveToDb | DelFiles)
fork(Producer.merge(tighteningData) | SaveToFile | ListFiles | SaveToDb | DelFiles)

[louthy]

Queue is quite badly named I think, it's closer to a Subject in Rx.NET. I will fix it to work correctly as a queue, and maybe add a new type that acts more like a Subject.

Interestingly, Producer.merge(...) does actually use a ConcurrentQueue internally to make sure that any merged Producers turn into a single stream. So, it does do exactly what you need (when 'merging' a single Producer).

[louthy]

@mozesa This should be fixed in v4.2.9. I've done minimal testing on it so far but it seems to be correct (the changes are pretty simple, so I don't foresee any issues); but if you can let me know how you get on that'd be good 👍 .

[mozesa]

@louthy I cannot believe. I really appreciate your effort. It works like a charm. I have implemented my own DatabaseIO and traits too.

You have mentioned somewhere here that now you actually enjoy more programming in C# than F# (thanks to language-ext), so now there is point (left) to learn F#?

So far I really enjoy working with (your opinionated) Aff and Eff, I don't know if it makes sense to mix and match F# with C#?
Btw, is it possible to use language-ext in F# (b/c I can imagine a way, when the BL is implemented in F# and the outermost layer (like UI) in C#)?

Anyway, I have to stick to .NET ecosystem.

I can't thank you enough.

[louthy]

Disclaimer: Remember the internet is full of people with opinions, my thoughts re: F# is just that: opinion. Please take it with a pinch of salt, and note I'm probably completely wrong, and I almost certainly have some bias because I built this library.

Having got the disclaimer out of the way ...

I would always encourage people to try and discover new languages, it will always help broaden the horizons. However, what I see with F# is simply a reskinned C#, but with significantly poorer tooling. The tooling in C# is best in class, and so in terms of productivity I'd want some major advantage over C# to switch to F#.

The one big advantage F# has over C# is it has discriminated-unions (with completeness checking). This is very valuable for functional programming. There's support for DUs in LanguageExt.CodeGen - just not completeness checks (which I am also working on for the next version of of the code-gen lib that will use Source Generators).

And so, although F# has an edge there, it's not that big at the moment, and it's surmountable.

Another advantage of F# is type-providers - which are a pretty sweet idea, but I've just found I ended up fighting them, or needing external resources available when building, etc. Although they're more principled than C#'s Source Generators, it's now possible to do what type-providers do in C#.

So, what's the downsides of F#:

• Interop - is poor. The idea that option in F# becomes the horrifically ugly FSharpOption in C# is awful. Especially as FSharpOption<A>.None is null. I mean wow, what a massive mistake. Take a look at how FSharpOption is converted to a language-ext Option:

public static Option<T> fs<T>(FSharpOption<T> fsOption) =>
    FSharpOption<T>.get_IsSome(fsOption)
        ? Some(fsOption.Value)
        : None;

Having FSharp* littered all over your C# is really poor. To the point that we convert to C# native type. Which you may think makes everything less problematic. But it only adds to the pain of sharing code across the boundaries.

• Interop - FSharpList - as above
• Interop - FSharpSeq - as above
• Interop - FSharpMap - as above
• Interop - FSharpFunc - as above x 1000
• Interop - () the unit type is void - gaaaah!
• Interop - creating and working with the F# DU types in C# is really poor, the construction is very hard, and the positional semantics for the tuples is super hard to deal with.

The standard argument is to write mapping facades to the F# or C# code. But that to me just means interop is an afterthought. If F# can't do easy interop with its big brother, then it's not worth the hassle.

• Type-system - If you look at other ML derived languages, and F#'s closest relative OCaml, then you'll see extra features like parametric modules; or in Haskell: higher-kinded types. These give additional power over and above what the C# type-system can do. But F# will never get those features because of some desire for interop-parity with C#. Which is bizarre, the interop story is already poor, and F# stops itself having a unique-selling point (like higher-kinds) because of its desire to play nice with C#. And so, outside of DUs there's no type-system USPs.
• Library ecosystem - It was pretty small when I used it in anger a few years back. I'm not sure where it is now. But it was clear that the F# ecosystem leant quite heavily on the BCL, which is very OO and very much aimed at the C# ecosystem. So F# code seems to be a weird mix of OO and functional. Obviously, that's an argument you could put on this library too, although I am making a very clear effort to give the tools to disconnect from the OO way as much as possible. I'm not sure it's there in F#.

I think F# will always be in the shadow of C#. C# is gaining all of the features of F#, and has more of Microsoft's resource on it. Yes, it's annoying that it doesn't do better type-inference, yes it's annoying that it's pretty verbose at times. But it's also Microsoft's flagship language. Right now, the only benefits I see from F# is:

• More terse syntax (mostly because type inference is better)
• Type inference
• Better composition of functions
  • Can partially apply
  • Method-groups need to die in a fire in C# - or become a runtime concept that can be resolved late
• DU completeness checking

Outside of that it's a reskinned C# with few real killer advantages and a very real tooling problem.

Although we (my company) dipped our toe into the F# pool for a while (we wrote a few services using F#), we have decided there's little to gain. Interestingly, we're now focussing a lot of effort on Haskell for our new projects. Haskell gives very real benefits over C#, which makes the poorer tooling OK (i.e. we are willing to put up with poorer tooling for a significantly better language).

Here's the disclaimer again...

Disclaimer: Remember the internet is full of people with opinions, my thoughts re: F# is just that: opinion. Please take it with a pinch of salt, and note I'm probably completely wrong, and I almost certainly have some bias because I built this library.

[mozesa]

Thank you so much for the detailed explanation.