Microservices – What is it. Going Faster with them

There are some important Principles of microservices(defining microservices).

https://www.youtube.com/watch?v=wgdBVIX9ifA – Martin Fowler

Whom to call a microservice –

Component Design – Libraries and Services.
Organised around the BUSINESS Capabilities and not the technologies. These teams should be directly responsible right upto the end user experience.
Having their own DATA boundaries. Every service should have their own database.Even SOA had different services but the same DB source. Doing this will separate schema and make them free to choose the data source they see correct.
They should have smart endpoints, unlike the ESB.
Monitoring is a must in order to cope up with service failures.
Infrastructure Automation is needed since there can be multiple types of systems that the services can interact with or created with. Continuous Devlivery etc are mandatory.
Build for failure. The remote service will fail. Similar to Netflix which makes it resilient.
Multiple versions acceptable. Self – monitoring. Loosely Coupled

Team size: Pizza sized teams.
_______________
– When to use & what are the advantages.
The microservices preserve modularity but make refactoring difficult.
Partial Deployment can be easily achieved in Microservices unlike monolith applications.
They are highly available but lack consistency.
They are platform independent.
_______________
– Must have if you are adopting microservices.
Rapid Provisioning of resources
Monitoring
Rapid Deployment
Have DevOps Culture

 

Challenges in Implementing Microservices:

https://www.youtube.com/watch?v=yPf5MfOZPY0 – Fred George
http://gotocon.com/amsterdam-2015/presentation/Challenges%20in%20Implementing%20MicroServices

The need of microservices – To Go Faster.

The organizations and startups want to move faster and with so many tools available today, like Docker, Cassandra, NoSql and terminologies and culture shift like MVP, Cloud, DevOps, these contribute in creating a pattern of development that helps move things faster and increase the rate of development.

Why ?
There are different types of problems and there is a lot of competition in the market.
Challenge 1.Deconstructing the database.
-Fortune 100 eg- Entity oriented, consistency essential.

As per Microservices: Event Buses to be used to replace operational databases.
DB per microservice if persistence is required. Multiple options for FBs
Few, writeable or transactional.

Rapids, Rivers and Ponds:
Rapids – Every Event. (Every message at every place – even the logs) in the same stream of information. Traffic stats, Service Stats.
Rivers – Themed Events : Reports etc.
Ponds – State/History, Events or Entities

This requires a high performance event bus. KAFKA. bus behind LinkedIn
250000 msgs per sec. It is a dumb bus. Also matters how many things we attach in the bus, which will decrease the number of events required.

We can attach certain message and then other messages for the rivers and use RabbitMQ.

We then attach the services to these events.
Handing Of messages to services based on corresponding traits.
We listen to a river but we always publish it to the rapids.
People might not see it if we push it to Rivers.

This helps us to build incremental applications.

a) There are event publishers and Solution Collectors. These communicate the event and get the response or solution of the event. Redis will serve as the db and cache it.

Challenge 2: Synchronous vs Asynchronous
Sync service – Turning module into services and calling them restfully.
Async – gives robustness, A/B testing can be done

Need Async Pattern : Need and Solution and working with the event bus.

Multiple Service Taxonomy depending on the client – timeout, web sockets, batch jobs. The processes can be of type – Soltion services, solution blockers, Client Biases, Enrichment Services.
And above all, everything is getting logged along with monitoring and KPIs

Chalenge 3 – Microservices or Clojure (Functional Programming)-
Microservices are like OO. Has its own persistence.
Closures – They love shared data and can only make new data can change old one. Eg. The UI of DailyMail from 130000 lines of Java to 4000 lines of closure. They have a big json structure of the page and it creates the html structure of the page. These are not reqd to make as a services.

Examples.
a) Ruby – frontend, Node for monitoring, Closure – heavy lifting and monitoring. Analysis with R.

Coupling – DB, Rest, Kafka

b) DailyMail :
Clojure – 3 services.
Node.js – dozens of node js(with avg size of 14 lines of code)
Rabbit –

c) Outpace : Clojure – 25
Orchestrations, RabbitMQ

Solution : Elixir – Functional programming languages built on Erlanf VM.

Challenge 4- Choosing Architectures and Frameworks: Too early to tell

Netflix frameworks. Pigato (based on ZeroMQ),

Challenge 5 – No Design Patterns yet.

Challenge 6 – Corequisite Technology, Processes, Organization structure. Dev Ops concepts, Mangerless processes.

Microservices – Implementations

Spotify –

Requirements:
Scale to millions of users.
Platform independent.
Complex business rules.
Innovate fast and respond to competitions.

How they do it – Autonomous full-stack teams.
spotify.github.io/apollo/ – writing services that focus on composability, simplicity, High performance, modern Java Idioms and features.

Pain Points of microservices – documentation. Harder to monitor.
Create increased latency since all the services are calling other services – View Aggregation services to solve that. (Idea from Netflix)

Netflix –

Next generation monitoring platform – atlas architecture and need.
http://techblog.netflix.com/2014/12/introducing-atlas-netflixs-primary.html

protobuf – binary serialization formats. Image sizes, prefetch requests. close tcp connections (keep-alive is kept on)  https://www.youtube.com/watch?v=ucK4FL0KKB4

UI Engineering – https://plus.google.com/+NetflixUIEngineering

Linux commands for search

To search a gzipped file:
zcat mobileapi-29.txt.tar.gz | cut -d ‘ ‘ -f 7,11 –output-delimiter=’      ‘ | egrep -a -oe ‘mobileapi/api_app_session.php(.)* ‘  -oe ‘<(.)*>’

– Filter based on egrep condition.
zcat mobileapi-29.txt.tar.gz | cut -d ‘ ‘ -f 7,11 –output-delimiter=’      ‘ | egrep -a -e ‘mobileapi/api_app_session.php(.)* ‘

(www.thegeekstuff.com/2009/05/zcat-zless-zgrep-zdiff-zcmp-zmore-gzip-file-operations-on-the-compressed-files/)

– Find with print and  xargs  find … -print 0 | xargs

zcat, awk, egrep, grep, zgrep.

Swift 2.2 & 3.0 Features and Proposals

https://github.com/apple/swift-evolution
Swift 3.0

– Requiring Leading Dot Prefixes for Enum Instance Member Implementations:  This is done to remove confusion with different standards at different places.
https://github.com/apple/swift-evolution/blob/master/proposals/0036-enum-dot.md
https://github.com/apple/swift-evolution/blob/master/proposals/0045-scan-takewhile-dropwhile.md
– Features related to Swift Project Manager  ! Long Proposal
https://github.com/apple/swift-evolution/blob/master/proposals/0082-swiftpm-package-edit.md
https://github.com/apple/swift-evolution/blob/master/proposals/0080-failable-numeric-initializers.md
https://github.com/apple/swift-evolution/blob/master/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md
https://github.com/apple/swift-evolution/blob/master/proposals/0025-scoped-access-level.md
https://github.com/apple/swift-evolution/blob/master/proposals/0052-iterator-post-nil-guarantee.md
https://github.com/apple/swift-evolution/blob/master/proposals/0060-defaulted-parameter-order.md

Swift 2.2 features: https://swift.org/blog/swift-2-2-released/
https://github.com/apple/swift-evolution/blob/master/proposals/0014-constrained-AnySequence.md
https://github.com/apple/swift-evolution/blob/master/proposals/0014-constrained-AnySequence.md
Providing some default tuple comparison.
– Naming Functions with Argument Labels in case of function overloading
https://github.com/apple/swift-evolution/blob/master/proposals/0021-generalized-naming.md
– typealias and associatedtype
Associated type now use the associatedtype keyword and typealias is available for implementing true associated type aliases. Earlier typealias keyword was used outside of protocols to declare type aliases and in protocols to declare associated types. Now it looks like this.
https://github.com/apple/swift-evolution/blob/master/proposals/0092-typealiases-in-protocols.md
https://github.com/apple/swift-evolution/blob/master/proposals/0011-replace-typealias-associated.md

protocol Sequence {
  associatedtype Iterator : IteratorProtocol
  typealias Element = Iterator.Element
}

typesaliases refers to Generic Type Aliases. It allows type aliases to introduce type parameters, which are in scope for their definition.
https://github.com/apple/swift-evolution/blob/master/proposals/0048-generic-typealias.md

– Expanding Swift Self to class members and value types
Self.dynamicType -> ‘Self’ to as shorthand for referring to the containing type (in the case of structs, enums, and final class) or the dynamic type (in the case of non-final classes).
https://github.com/apple/swift-evolution/blob/master/proposals/0068-universal-self.md

 

Swift Language – A Starter

Guidelines laid down for Swift: Which will be part of the Swift 3.0 to be launched by end of December 2016.

Some of the important go to design guidelines laid down by the Swift team [IMP].
https://swift.org/documentation/api-design-guidelines/

Swift Evolution: Tracking the changes and updates
https://github.com/apple/swift-evolution/blob/master/README.md

Continuous Integration – via Jenkins.

Package Manager: https://swift.org/package-manager/#conceptual-overview

There are only 2 types of decisions

Some decisions are consequential and irreversible or nearly irreversible – one-way doors – and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don’t like what you see on the other side, you can’t get back to where you were before. We can call these Type 1 decisions. But most decisions aren’t like that – they are changeable, reversible – they’re two-way doors. If you’ve made a suboptimal Type 2 decision, you don’t have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high judgment individuals or small groups.

As organizations get larger, there seems to be a tendency to use the heavy-weight Type 1 decision-making process on most decisions, including many Type 2 decisions. The end result of this is slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention. We’ll have to figure out how to fight that tendency.

– Jeff Bezos

Golang Concepts – mapped with OS concepts

Golang has been developed with a lot of thinking about how developers like to develop. Hence, there are a lot of concepts and background reason for every decision made by them. We just need to go through the documentation to understand them.

I will be listing some of the sections that are worth the read :
– The go memory model: https://golang.org/ref/mem
The section on Synchronization talks about memory initialization, memoty handling in goroutines, channel communication and locks.

– Memory allocation with new & make:
https://golang.org/doc/effective_go.html#allocation_new

– Concurrency:
https://golang.org/doc/effective_go.html#concurrency
Discussion about sharing via communcation, goroutines, channels, Parallelization

=========BASICS========
———-TYPES———-
– Variable Declarations:
Outside a function, every statement begins with a keyword (var, func, and so on) and so the (Short Variable Declaration) := construct is not available. (https://tour.golang.org/basics/10)

– Types: All defines types and their alias defined. (https://tour.golang.org/basics/11)
bool string
int = int8(also byte) int16 int32(also rune) int64
uint = uint8 uint16 uint32 uint64 uintptr
float32 float64
complex64 complex128
The int, uint, and uintptr types are usually 32 bits wide on 32-bit systems and 64 bits wide on 64-bit systems.
Zero values are the default values set for the types: 0, false, “”, nil

– Constants :
Constants are declared with const keyword and cannot be declared using the := syntax. Numeric constants are high-precision values.

– Type Conversion:
Unlike in C, in Go assignment between items of different type requires an explicit conversion. There is no auto conversion. (https://tour.golang.org/basics/13)

———-ITERATIONS——–
– FOR: It has three parts where 1st and 3rd are optional. Also, FOR is Go’s while after dropping semicolons and the 1st and 3rd condition.
There are no parentheses surrounding the three components of the for statement and the braces { } are always required. Same for IF statements.
At that point you can drop the semicolons: C’s while is spelled for in Go.

-IF: Like for, the if statement can start with a short statement to execute before the condition. Variables declared by the statement are only in scope until the end of the if.

-Switch: Switch can also have a short assignment. Also, the value to be compared can be missing in which case, the Case can have a comparison which would return true/false
A case body breaks automatically, unless it ends with a fallthrough statement.

————-ADVANCED TYPES————
Pointers, Structs, Pointers to Structs,

Struct Literals(instance): You can list just a subset of fields by using the Name: syntax. (And the order of named fields is irrelevant.)
Arrays: [n]T An array’s length is part of its type, so arrays cannot be resized. Printing an array without an index will print all values.

Slices: A slice is dynamically sized. A slice gives much more flexibility. Eg: arr[0:5] will give a slice of first 5 elements of an array. Slices are like reference to array. It changes the underlying structure of the array it references.
Slice Defaults – 0 for low bound and length for upper bound. The upper bound takes the (upper value -1)  (https://tour.golang.org/moretypes/10)
A slice has both a length and a capacity. It goes out of bounds if the length goes over capacity. (https://tour.golang.org/moretypes/11)
s = s[:2] => lower bound will take values from the current slice, Upper Bound takes value from the original slice. Zero value of slice is nil which has len = cap = 0 and no underlying array.  Slices of slices can also be made easily.
(More Usage and Internals: https://blog.golang.org/go-slices-usage-and-internals)

Make: Used to inititialize and allocate memory. Takes optional parameters. It takes type, length and cap. The make function returns a map of the given type, initialized and ready for use. (https://tour.golang.org/moretypes/13)

append: (https://tour.golang.org/moretypes/15)
range: For Iterating collections. When ranging over a slice, two values are returned for each iteration. The first is the index, and the second is a copy of the element at that index.  And if you only want the index, drop the “, value” entirely. (https://tour.golang.org/moretypes/16)

Maps: If the top-level type is just a type name, you can omit it from the elements of the literal.  eg.  type Vertex struct {Lat, Long int64}
var m = map[string]Vertex{
“Bell Labs”: {40.68433, -74.39967},
“Google”:    {37.42202, -122.08408},
}
Map operations :
delete(m, key) , checking existence with  elem, ok = m[key]

Function Values: Functions are values too. and can be passed around like values. Function values may be used as function arguments and return values.  (https://tour.golang.org/moretypes/24)

Function closures: In Go, functions may be closures too. A closure is a function value that references variables from outside its body. Eg.
return func(x int) int {
sum += x
return sum
}

=========METHODS & INTERFACES========
A method differs from go as a method has a receiver defined. A method can be defined on a non-struct type as well. The receiver in such a case should be defined in the same package as the method.

Pointers:
Functions with a pointer argument must take a pointer while methods with pointer receivers take either a value or a pointer as the receiver when they are called. Also, methods with value receivers take either a value or a pointer as the receiver (https://tour.golang.org/methods/7) .
(to provide a pointer we use &<variable>). To define a pointer as an input, we mention var1 *type which specifies that the var1 is a reference to the pointer.

Interfaces: An interface type is defined as a set of method signatures. A value of interface type can hold any value that implements those methods. And it matters whether the receiver type that implements the interface is a struct or *struct. Interfaces are implemented implicitly. There is no explicit declaration of intent, no “implements” keyword.
Interface values : Under the covers, interface values can be thought of as a tuple of a value and a concrete type: (value, type). Use describe to find it. Note that an interface value that holds a nil concrete value is itself non-nil. (https://tour.golang.org/methods/11)
Empty Interface: The interface type that specifies zero methods is known as the empty interface. Empty interfaces are used by code that handles values of unknown type. For example, fmt.Print takes any number of arguments of type interface{}.

Type Assertions: A type assertion provides access to an interface value’s underlying concrete value.  t, ok := i.(T) .
Type Switch: This can be used in type switches wherever required.  switch v := i.(type)
Stringer:  Stringer is a interface type that can describe itself as a string. It has the String function declared. (https://tour.golang.org/methods/18)

Error: Errors are expressed by error interface type (similar to Stringer, error interface looks for Error() string function) (https://tour.golang.org/methods/20)

Readers: io. Reader interface is used for reading which has a Read function to read the end of stream of data.  (https://tour.golang.org/methods/21 https://tour.golang.org/methods/22). There are also ways in which a reader wraps another reader.
There are various implementation of it including files, network connections, compressors, ciphers, and others. (https://golang.org/search?q=Read#Global)

Images: Package image defines the Image interface:  ColorModel(), Bounds(),  At(x, y int) functions. The color.Color and color.Model types are also interfaces which have default implementations  like color.RGBA and color.RGBAModel.

=========CONCURRENCY========
Goroutines: Goroutines run in the same address space, so access to shared memory must be synchronized. The sync package provides useful primitives.

Channels: Channels are a typed conduit through which you can send and receive values with the channel operator, <-.  It is defined by chan and must be initialized using make (similar to slices, maps). By default, sends and receives block until the other side is ready. This allows goroutines to synchronize without explicit locks or condition variables.  (https://tour.golang.org/concurrency/2)

Buffered Channels: Channels can be buffered. Provide the buffer length as the second argument to make to initialize a buffered channel. It sends to a buffered channel block only when the buffer is full. Receives block when the buffer is empty.

Closing a channel : Channel can be closed using close(ch) only by a Sender (and never a receiver ). Channels do not require to be closed. It is only necessary when the receiver needs to be told of that no more values are coming.

Select: Waiting on gorountine is achieved using this.  The select statement lets a goroutine wait on multiple communication operations. A select blocks until one of its cases can run, then it executes that case. It chooses one at random if multiple are ready.

Default Select: The default case in a select is run if no other case is ready.

Locks in GO: Locks can be achieved using sync.Mutex and its two methods Lock and Unlock (https://tour.golang.org/concurrency/9)
++++++++++++++++++++++++++++++++++
Useful Links: https://tour.golang.org/concurrency/11
The further reads from Go Blog:
https://blog.golang.org/json-and-go
https://blog.golang.org/defer-panic-and-recover
https://blog.golang.org/error-handling-and-go
https://blog.golang.org/laws-of-reflection
https://blog.golang.org/organizing-go-code

https://blog.golang.org/profiling-go-programs
https://blog.golang.org/share-memory-by-communicating
https://blog.golang.org/go-concurrency-patterns-timing-out-and
https://blog.golang.org/first-class-functions-in-go-and-new-go

https://blog.golang.org/introducing-gofix
https://blog.golang.org/json-rpc-tale-of-interfaces
https://blog.golang.org/gif-decoder-exercise-in-go-interfaces