reF#ACTORing using F# and ACTOR model Vagif Abilov

slide1. reF#ACTORingusing F# and ACTOR model Vagif Abilov<br>
slide3. About myself Work in Norwegian company Miles
Mail: vagif.abilov@gmail.com
Twitter: @ooobject
GitHub: object
Blog: http://vagifabilov.wordpress.com/
Articles: http://www.codeproject.com
Maintainer of Simple.OData.Client, contributor to few other open source projects<br>
slide5. Our project MediaDistributionEngine<br>
slide6. 1 2 3 4 5 6<br>
slide7. How was media distribution managed earlier? First generation application was developed in 2010-2012
Very well written in C# with respect to SOLID principles
.NET 4.0
Oracle Entity Framework provider (Devart dotConnect)
RabbitMQ
Rebex FTP library
StructureMap
Does it really need to be rewritten?<br>
slide8. Agenda – motivation line Why rewrite?
Why actor model and Akka.NET?
Why F#?<br>
slide9. Agenda – main lessons learned Akka.NET F# API and finite state machines
”Tell, don’t ask” means ”Tell and be told”
How we learned to stop worrying and love actor supervision
Neither SQL, nor NoSQL: persistent actors to the rescue<br>
slide10. Agenda – reF#actoring example Rewriting C# scheduler actor in F#<br>
slide11. Agenda – topics left out Planning actor lifetime
Hosting actors
Combining actors with message queues
Debugging actors
Testing actors with Akka TestKit and FsCheck<br>
slide12. Agenda – obligatory disclaimer Actors are not silver bullets<br>
slide13. Why rewrite? Because F*ck You, That’s Why!

No, our choice wasn’t purely driven by technological interest<br>
slide14. Reasons to rewrite – functional changes Original system written for a single specific cloud storage provider, the new one should work with multiple storage providers
Different file upload protocols (FTP vs. HTTP)
Database polling to be fully replaced with message queues<br>
slide15. 1 2 3 4 5 6<br>
slide16. Reasons to rewrite – revised design goals Avoid manual thread management
Strict mutability control
Enforced component isolation
Replace general-purpose programming model with the one tailored for message processing
Better exploit functional transformations<br>
slide17. An actor model in 1... well... 3 minutes An actor can do:
Send messages to other actors (and receive messages from others)
Handle one message at a time
Change its own state
Create and supervise child actors
An actor doesn’t expose its state except the possibility to send a message that contains an indication of its current state
An actor rents a thread only for the duration of message processing<br>
slide18. / /user /system /user/a /user/b /user/c /user/b/1 /user/b/2<br>
slide19. / /user /system /user/a /user/b /user/c /user/b/1 /user/b/2<br>
slide20. Actors vs. traditional OOP objects You can give human names to class instances, but it won’t breathe life into them
Objects don’t act – they are acted upon
Whatever state or operation an object exposes can be used against him by any number of evil strangers acting simultaneously from evil threads<br>
slide21. Actors vs. traditional OOP objects An actor may spend most of its life in a deep long sleep, but when he decides to act it’s his conscious choice for what he carries full and sole responsibility
Good actors respect other actors privace and don’t ask them questions – instead they just tell them about their intentions so they may receive something back in return<br>
slide22. So why actor model and Akka.NET? If you want the reason in a single short word: TIRED!

What we planned to gain
No more explicit thread management
No more explicit shared state protection
Declarative migration of state: actors as finite state machines<br>
slide23. Why F#? Just say “no” to mutable variables!
Idiomatic F# API, actors as functions without any internal state
An actor in F# can be implemented as a finite state machine with each state managed by its own function
F# is made for functional transformations!
Less ceremony, more compact code<br>
slide24. The simplest actor written in F# let myActor (mailbox: Actor<_>) =  let rec loop () = actor {  let! message = mailbox.Receive() printfn "%A" message  return! loop ()  }  loop ()<br>
slide25. Akka.NET API and finite state machines let myActor (mailbox: Actor<_>) =  let rec disconnected () = actor {  let! message = mailbox.Receive() match message with  | Connect ->  let connection = CreateConnection () return! connected (connection) | _ ->  printfn "%A is invalid in disconnected state" message  return! disonnected () }  // Continues on the next slide<br>
slide26. Akka.NET API and finite state machines // … continues and connected (connection) = actor {  let! message = mailbox.Receive() match message with  | Disconnect ->  ReleaseConnection (connection) return! disonnected () | _ ->  printfn "%A is invalid in connected state" message  return! connected (connection) }  disconnected ()<br>
slide27. “Tell, don’t ask” means “tell and be told” The only way for actors to exchange information is via sending messages
Akka’s standard method to send message is Tell, but it also exposes a method Ask that suspends an actor until the message receives a reply
Akka.NET F# API defines operators <! and <? for Tell and Ask
Suspending an actor is a bad sign of non-reactive code: a good actor is asleep while waiting for messages
What if reply never comes?<br>
slide29. But wait – there’s more ASKperformanceisjust
HORRIBLE<br>
slide30. Let’s do some benchmarks Test machine: HP Spectre x360 i5 2.20 GHz, Windows 10
Time in seconds spent on 1 million operations

Void method invocation 0.003 (~300M/s)
Method invocation with result assignment 0.004 (~250M/s)<br>
slide31. Let’s do some benchmarks Test machine: HP Spectre x360 i5 2.20 GHz, Windows 10
Time in seconds spent on 1 million operations

Void method invocation 0.003 (~300M/s)
Method invocation with result assignment 0.004 (~250M/s)
Async Task method invocation with await 1.864 (~500K/s)
Async Task<int> method invocation with await 1.790 (~500K/s)<br>
slide32. Let’s do some benchmarks Test machine: HP Spectre x360 i5 2.20 GHz, Windows 10
Time in seconds spent on 1 million operations

Void method invocation 0.003 (~300M/s)
Method invocation with result assignment 0.004 (~250M/s)
Async Task method invocation with await 1.864 (~500K/s)
Async Task<int> method invocation with await 1.790 (~500K/s)
Actor Tell 0.233 (~4M/s)<br>
slide33. Let’s do some benchmarks Test machine: HP Spectre x360 i5 2.20 GHz, Windows 10
Time in seconds spent on 1 million operations

Void method invocation 0.003 (~300M/s)
Method invocation with result assignment 0.004 (~250M/s)
Async Task method invocation with await 1.864 (~500K/s)
Async Task<int> method invocation with await 1.790 (~500K/s)
Actor Tell 0.233 (~4M/s)
Actor Ask 6.395 (~150K/s)<br>
slide34. Why is Ask so slow? Details: http://bartoszsypytkowski.com/dont-ask-tell-2/
Since an actor can only receive information from messages in its mailbox, then how will it receive a response to its request?
Scan the entire mailbox and extract the response message – breaks the sequential mailbox message processing rule, complicates mailbox implementation
Create a temporary actor that would act as a listener for that response message
Akka chooses 2nd approach that results in severe performance degradation of scenarios involving Ask<br>
slide35. But what if we need a response from an actor? Integrate response retrieval in the actor’s switchable behaviours (using Become method in C# or actor functions in F#)
While the actor is in awaiting response state, it will need to Stash messages that requires the awaited response for its processing
If the actor’s mailbox is empty, it will sleep until the response message is received, making it a good reactive software citizen
Once the response message comes, the actor invokes UnstashAll to bring back stashed messages and changes its state so it no longer awaits for the response<br>
slide36. Example 1: replacing Ask with response aggregation - before let rec loop () =
actor {
let! message = mailbox.Receive () match message with
| Process item ->
let (details : ItemDetails) =  actorA <? GetItemDetails item |> Async.RunSynchronously actorB <! UseItemDetails item details

return! loop ()
}<br>
slide37. Example 1: replacing Ask with response aggregation - after let rec idle () =
actor {
let! message = mailbox.Receive () match message with
| Process item ->
actorA <! GetItemDetails item return! awaiting_details (item)

return! idle ()
}
 // Continues on the next slide<br>
slide38. Example 1: replacing Ask with response aggregation - after // … continues   and awaiting_details (item) =
actor {
let! message = mailbox.Receive () match message with
| ItemDetails details ->
actorB <! UseItemDetails item details mailbox.UnstashAll()  return! idle () | _ ->  mailbox.Stash()
return! awaiting_details () }<br>
slide39. Example 2: collecting responses from 1000000 hotels // … skipped the rest of the implementation   and collecting (responses : Response list) =
actor {
let! message = mailbox.Receive () match message with
| Response response ->
return! collecting (response :: responses) | Enough ->
mailbox.UnstashAll()  return! idle () | _ ->  mailbox.Stash()
return! collecting (responses) }<br>
slide40. Extended benchmarks Test machine: HP Spectre x360 i5 2.20 GHz, Windows 10
Time in seconds spent on 1 million operations

Void method invocation 0.003 (~300M/s)
Method invocation with result assignment 0.004 (~250M/s)
Async Task method invocation with await 1.864 (~500K/s)
Async Task<int> method invocation with await 1.790 (~500K/s)
Actor Tell 0.233 (~4M/s)
Actor Ask 6.395 (~150K/s)
Actor Tell and aggregate responses via Become 0.884 (~1M/s)<br>
slide41. So when can an actor Ask? “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live” John F. Woods on usage of comma operator (C++ language forum, 1991)
In a similar way and with a few exceptions it’s advisable to treat every actor that is going to respond to an Ask operation as a person suffering from manic depressive disorder
No, this is not an emotional advise, it’s purely practical<br>
slide42. OK, but when can an actor Ask? Some valid use cases Obtaining IActorRef for an actor
Requesting response to Identify message
Requesting state for a persistent actor
Querying actors with trivial implementation and without dependencies on external resources (but ask yourself first: do you need such trivial actors?)<br>
slide43. How we learned to stop worrying and love actor supervision Resilience
the ability of a substance or object to spring back into shape, elasticity
the capacity to recover quickly from difficulties, toughness

Supervision
the act or function of overseeing something or somebody

Prop
formally known as (theatrical) property, is an object used on stage or on screen by actors during a performance or screen production<br>
slide44. Error handling in C# connection.ExecuteCommand(cmd);<br>
slide45. Error handling in C# try{  connection.ExecuteCommand(cmd);}catch (SqlException ex) {  throw new ProductUpdateException("Unable to update product", ex);}<br>
slide46. Error handling in C# try{  connection.ExecuteCommand(cmd);}catch (SqlException ex) {  // Retry? // Recover database connection? // What to do with other places that use the same connection?}catch (Exception ex) { // Or should we catch other exception types?}<br>
slide47. Error handling in C# queue.Subscribe(x => HandleMessage(x));void HandleMessage(QueueMessage msg){
try { Process(msg); msg.Ack();  } catch (QueueException ex)  {  // Reconnect? Subscribe? BTW, we are in a different thread }}<br>
slide48. Exception handling considered harmful Article by Jason Robert Carey Patterson
http://www.lighterra.com/papers/exceptionsharmful

”Exception handling introduces a hidden, "out-of-band" control-flow possibility at essentially every line of code.”

”Exception handling does not fit well with most of the highly parallel programming models currently in use or being explored (fork/join, thread pools and task queues, the CSP/actor model etc), because exception handling essentially advocates a kind of single-threaded "rollback" approach to error handling, where the path of execution – implicitly a single path – is traversed in reverse by unwinding the call stack to find the appropriate error handling code.”<br>
slide49. Enter actor model Initial approach to queue actor messages
type QueueCommand =
| Connect of QueueDetails | Disconnect | Subscribe of IActorRef | Unsubscribe | Publish of QueueMessage | Receive of AckId * Payload | Ack of AckId | Nack of AckId<br>
slide50. Queue actor (initial revision) let queueActor (mailbox: Actor<_>) =
let rec disconnected () =
// … skipped the implementation
and connected (connection : IConnection) = // … skipped the implementation
and subscribed (connection : IConnection, subscriber : IActorRef) =
// … skipped the implementation disconnected ()<br>
slide51. But what about error handling? If an exception is not caught in actor code, it will be managed by actor’s supervisor who will apply configured supervision strategy
Supervision strategy may apply only to the failing actor (one-for-one) or to all child actors of the supervisor (all-for-one)<br>
slide52. Standard supervision strategies A supervisor may take one of the following actions when dealing with actor’s exceptions:
Restart the actor (default)
Resume the actor
Stop the actor
Escalate<br>
slide53. Example of a supervision strategy let strategy () =  Strategy.OneForOne((fun ex -> match ex with  | :? ArgumentNullException ->  Stop | :? ArgumentOutOfRangeException ->  Restart | :? ArgumentException ->  Resume | _ ->  Escalate), 3, TimeSpan.FromSeconds(10.))<br>
slide54. Applying default supervision strategy Exception occurs in actor code
Actor suspends execution
Supervisor is notified about failing actor and exception details
Supervisor disposes the failing actor but saves its mailbox
Supervisor creates a new instance of an actor using same actor props and assigns it the mailbox saved in the previous step<br>
slide55. Revisiting queue actor commands type QueueCommand =
| Connect // sent with props | Disconnect | Subscribe of IActorRef | Unsubscribe | Publish of QueueMessage | Receive of AckId * Payload | Ack of AckId | Nack of AckId<br>
slide56. Revisiting queue actor commands type QueueCommand =
| Connect // sent with props | Disconnect // no longer needed  | Subscribe of IActorRef | Unsubscribe | Publish of QueueMessage | Receive of AckId * Payload | Ack of AckId | Nack of AckId<br>
slide57. Revisiting queue actor commands type QueueCommand =
| Connect // sent with props | Disconnect // no longer needed, stop the actor  | Subscribe of IActorRef // sent with props  | Unsubscribe // no longer needed, stop the actor  | Publish of QueueMessage | Receive of AckId * Payload | Ack of AckId | Nack of AckId<br>
slide58. Queue actor (revised) let queueActor (queueDetails: QueueDetails) (subsriber: IActorRef) (mailbox: Actor<_>) =
let connection = factory.CreateConnection(queueDetails) // … subsribe subscribers
 let rec loop () =
// … skipped the implementation
 loop ()<br>
slide59. Spawning a queue actor let queue = spawn system "queues.file_upload" queueActor  queueDetails subscriber

queueDetails - PROP
subscriber - PROP<br>
slide60. Neither SQL, nor NoSQL: persistent actors to the rescue Akka actor persistence is based on principles of event sourcing
Greenfield projects have an advantage of mapping all persistent states to actors
Some of such actors will only deal with persistent state management
In our project we don’t mix functions of doer-actors and bookkeeper-actors
This is the first time in my professional life when we ran the whole backend project without defining a single database table<br>
slide61. Persistent state example type FileDistribution = { StorageProvider : FileStorageProvider  Locator : RemoteLocation  CdnPath : AbsoluteUrl option  GeoRestriction : GeoRestriction option  AccessLevel : AccessLevel option  State : DistributionState  Length : uint64  Timestamp : DateTimeOffset }
 type DistributionLocatorCommand =  | AssignLocator of FileDistribution  | RemoveLocators  | QueryLocators  | TakeSnapshot<br>
slide62. Currently supported Akka.NET persistence backends In-memory + local files
Microsoft SQL Server
Azure tables
PostgreSql
Cassandra
MySql
MongoDB
Redis
Sqlite
Oracle<br>
slide63. Akka.NET event journal CREATE TABLE EventJournal ( Ordering BIGINT IDENTITY(1,1) PRIMARY KEY NOT NULL, PersistenceID NVARCHAR(255) NOT NULL, SequenceNr BIGINT NOT NULL, Timestamp BIGINT NOT NULL, IsDeleted BIT NOT NULL, Manifest NVARCHAR(500) NOT NULL, Payload VARBINARY(MAX) NOT NULL, Tags NVARCHAR(100) NULL CONSTRAINT QU_EventJournal UNIQUE (PersistenceID, SequenceNr))<br>
slide64. reF#actoring: rewriting a C# actor in F# Akka comes with only in-memory message scheduler
Quartz.NET (port of a Java library) is a full-featured open source job scheduling system supporting both in-memory and ADO.NET backends
Akka.Quartz.Actor is an job scheduling actor written in C# and exposing Quartz.NET features (https://github.com/akkadotnet/Akka.Quartz.Actor)
Let’s rewrite it in F# (we even had a reason!)<br>
slide65. Akka.Quartz.Actor files and lines QuartzActor.cs 97 93 (without comments)QuartzJob.cs 35 32Commands/IJobCommand.cs 9 6Commands/CreateJob.cs 33 21Commands/RemoveJob.cs 26 17Events/IJobEvent.cs 13 10Events/JobEvent.cs 23 17Events/JobCreated.cs 36 30Events/CreateJobFail.cs 26 20Events/RemoveJobFail.cs 23 20

10 files, 2 interfaces, 8 classes, 321 lines of code (267 without comments)<br>
slide66. CreateJob command public class CreateJob : IJobCommand { public CreateJob(IActorRef to, object message, ITrigger trigger) { To = to;  Message = message; Trigger = trigger; } public IActorRef To { get; private set; } public object Message { get; private set; } public ITrigger Trigger { get; private set; }}<br>
slide67. Actor’s message handler public class QuartzActor : ActorBase{ // … Skipped the rest of the implementation protected override bool Receive(object message) {
return message.Match() .With<CreateJob>(CreateJobCommand) .With<RemoveJob>(RemoveJobCommand) .WasHandled; } // … Skipped the rest of the implementation}<br>
slide68. Responding to CreateJob command if (createJob.To == null){ Context.Sender.Tell(new CreateJobFail(null, null,  new ArgumentNullException("createJob.To")));}else if (createJob.Trigger == null){ Context.Sender.Tell(new CreateJobFail(null, null,  new ArgumentNullException("createJob.Trigger")));
}else{
// … The real stuff is coming on the next slide}<br>
slide69. Responding to CreateJob command try{ var job = QuartzJob.CreateBuilderWithData(createJob.To, createJob.Message) .WithIdentity(createJob.Trigger.JobKey) .Build(); _scheduler.ScheduleJob(job, createJob.Trigger); Context.Sender.Tell(new JobCreated(createJob.Trigger.JobKey, createJob.Trigger.Key));}catch (Exception ex){ Context.Sender.Tell(new CreateJobFail(createJob.Trigger.JobKey, createJob.Trigger.Key, ex));}<br>
slide70. Executing scheduled job public class QuartzJob : Ijob{ private const string MessageKey = "message"; private const string ActorKey = "actor";
 public void Execute(IJobExecutionContext context) { var jdm = context.JobDetail.JobDataMap; if (jdm.ContainsKey(MessageKey) && jdm.ContainsKey(ActorKey)) { var actor = jdm[ActorKey] as IActorRef; if (actor != null) { actor.Tell(jdm[MessageKey]); } }
}
}<br>
slide71. Entering F# Single file Schedule.fs
2 type aliases
2 discriminated unions
1 private type
1 actor function
85 lines of code (no comments)

Remember C#? 10 types (2 + 8), 267 LOC<br>
slide72. F# schedule actor open Systemopen Akka.Actortype JobMessage = obj type JobId = obj type JobCommand = | CreateJob of IActorRef * JobMessage * DateTimeOffset | RemoveJob of JobId type JobCommandResult = | Success of JobId | Error of JobId * Exception

C# project contained 8 files with command/event definitions (141 lines)<br>
slide73. F# schedule actor type private QuartzJob () =  static let MessageKey = "message"  static let ActorKey = "actor"  interface IJob with  member this.Execute (context : IJobExecutionContext) =  let jdm = context.JobDetail.JobDataMap  if jdm.ContainsKey(MessageKey) && jdm.ContainsKey(ActorKey) then  match jdm.[ActorKey] with  | :? IActorRef as actor -> actor <! jdm.[MessageKey]  | _ -> ()
 static member CreateBuilderWithData (actorRef : IActorRef, message : obj) =  let jdm = new JobDataMap()  jdm.AddAndReturn(MessageKey, message).Add(ActorKey, actorRef)  JobBuilder.Create<QuartzJob>().UsingJobData(jdm)<br>
slide74. F# schedule actor let scheduleActor props (mailbox: Actor<_>) =  let t = TriggerBuilder.Create().StartNow().Build() let scheduler =  match props with  | Some props -> StdSchedulerFactory(props).GetScheduler()  | None -> StdSchedulerFactory().GetScheduler()  scheduler.Start()  mailbox.Defer (fun _ -> scheduler.Shutdown()) let rec loop () =  actor {  let! message = mailbox.Receive () // … Message matching is shown on the next slide } loop ()<br>
slide75. F# schedule actor match message with | CreateJob (actor, message, scheduledTime) ->  match (actor, scheduledTime) with  | (null,_) -> mailbox.Sender()  <! Error (null, new ArgumentException("CreateJob actor is null"))  | _ ->  let trigger = TriggerBuilder.Create().StartAt(scheduledTime).Build()  try  let job = QuartzJob.CreateBuilderWithData(actor, message)  .WithIdentity(trigger.JobKey) .Build()  scheduler.ScheduleJob(job, trigger) |> ignore  mailbox.Sender() <! Success trigger.JobKey  with ex ->  mailbox.Sender() <! Error (trigger.JobKey, ex) | RemoveJob (jobKey) -> // … RemoveJob handler is skipped<br>
slide76. reF#actoring observations F# code is more compact than C#, and so are actor functions written in F#, in our case they take 60-70% fewer lines than its C# counterpart
F# discriminated unions is a perfect choice for definitions of actor messages (commands and events)
Fewer null checks, only concerning about nulls coming from outside
In most cases we put both actor function and all related type definitions in a single file<br>
slide77. reF#actoring metrics (whole project)<br>
slide78. Actors are not silver bullets Remember benchmarks? If all you need is direct method invocation, just do it!
Shallow stack trace not only complicates debugging, it also eliminates potential transactional boundaries - this can be a big loss in some scenarios
Actors are only for internal use – external systems don’t need to know they communicate with an actor system
Actors are not easily composable (Akka streams offer actor composition)<br>
slide79. So when is it OK to define actors? Parallel computation
State control via declarative behavior
Configurable scalability
Sufficiency of one-way communication<br>
slide80. And now for something completely different Goodbye $b

A song about an actor<br>
slide81. Akka and actor model concepts used in the song Actors process one message at a time
Actor instance is created or restarted using a set of props
Actor can be referred by its ActorRef
Akka routers use names $a, $b, $c etc. for actors in the pool
If an actor fails, it can be restarted in accordance to its supervision strategy, new actors instance gets the same name
If an actor in a pool fails and the supervision strategy is AllForOne + Restart, then all actors in a pool are restarted
A good practice is to use Tell command with actors, not Ask<br>
slide82. And now for something completely different Goodbye $b

A song about an actor<br>
slide83. Goodbye, $b
Though I never knew at all
They’d chosen to restart you
And new instance got your name
But when you were alive
You played your role according to your props
And every single message knew
That for you she’s only one<br>
slide84. And it seems to me
You lived your life
Like a good reactive code
Only using CPU cycles
When the message came
And I would've liked to have known you
But Supervisor’s fast
Your instance burned out long ago
Your ActorRef will last<br>
slide85. Loneliness was tough
You had to sleep alone most of the time
And when you were awakened
They’d let you tell but never ask
Even when you died
It was another sibling's fail
But supervision strategy
Was clear and brutal: AllForOne<br>
slide86. And it seems to me
You lived your life
Like a good reactive code
Only using CPU cycles
When the message came
And I would've liked to have known you
But Supervisor’s fast
Your instance burned out long ago
Your ActorRef will last<br>
slide87. Thank you! Work in Norwegian company Miles
Mail: vagif.abilov@gmail.com
Twitter: @ooobject
GitHub: object
Blog: http://vagifabilov.wordpress.com/
Articles: http://www.codeproject.com
Maintainer of Simple.OData.Client, contributor to few other open source projects<br>