- Golang 6
- Linux 5
- Kafka 4
- Kubernetes 3
- Distributed Computing 2
- Spark 2
- AWS 1
- C 1
- Cloud 1
- Redis 1
- Storm 1
Golang
Golang Notes
Golang is an open-source, compiled, and statically typed programming language designed by Google. It is built to be simple, high-performing, readable, and efficient.
Using Golang modules with CircleCI
An overview of using Golang modules with CircleCI, which is a continuous integration and delivery platform.
CGO - How to use C shared library code
I was working with CGO code and realized the need of keeping the C code in a shared library and using it from Go code. Here is a brief run down of how to go about it.
CGO - Bridge between C and Go
cgo allows Go and C programs to interoperate. This is very useful feature which lets C existing libraries to be called by Go. On the other hand Go libraries can also be called by C code.
GRPC Tutorial
I will be using Golang in this post. Let’s install the software needed. There are equivalent commands to try on Raspberry-PI.
Golang Testing
Testing a.k.a. automated testing, is the practice of writing small programs that check that the code under test behaves as expected for certain inputs, which are either carefully chosen to exercise certain features or randomised to ensure broad coverage.
Linux
Linux Virtualization
Virtualization solutions allow multiple operating systems and applications to run in independent partitions on a single computer. Using virtualization capabilities, one physical computer system can function as multiple “virtual” systems.
CGO - How to use C shared library code
I was working with CGO code and realized the need of keeping the C code in a shared library and using it from Go code. Here is a brief run down of how to go about it.
Extended Berkeley Packet Filter (eBPF)
eBPF stands for extended Berkeley Packet Filter but officially referred to as BPF. BPF was intially developed for high performance packet capture in 1992. In recent years (2012 - 2014), BPF is rewritten to a general purpose virtual machine that can be used for many things like networking, monitoring, security and performance analysis.
Netfilter and IPTables
iptables is a management tool for the firewall software netfilter in the Linux kernel. iptables is located in the user space while netfilter is located in the kernel space, where functionalities of firewalling, network address translation (NAT), packet content modification and packet filtering is implemented. Both together are commonly referred to just iptables.
Sockets
Sockets are the preferred way of network communication. They are provided by Operating Systems as socket API, which are based on the principles of reading and writing files.
Kafka
Kafka Producer and Consumer In Golang
Apache Kafka is a distributed publish-subscribe messaging system that is designed to be fast, scalable, and durable. Kafka stores streams of records (messages) in topics. Each record consists of a key, a value, and a timestamp. Producers write data to topics and consumers read from topics.
Patterns for Composite Containers
Building an application from modular containers means thinking about symbiotic groups of containers that cooperate to provide a service, not one container per service. In Kubernetes, the embodiment of this modular container service is a Pod. A Pod is a group of containers that share resources like file systems, kernel namespaces and an IP address. The Pod is the atomic unit of scheduling in a Kubernetes cluster, precisely because the symbiotic nature of the containers in the Pod require that they be co-scheduled onto the same machine, and the only way to reliably achieve this is by making container groups atomic scheduling units.
Kafka Connect and Streams APIs
Here is a typical architecture having Sources, Sinks, Connect Cluster, Kafka Cluster and Kafka Streams Applications.
Kafka Basics
Apache Kafka is a distributed publish-subscribe messaging system that is designed to be fast, scalable, and durable. Kafka stores streams of records (messages) in topics. Each record consists of a key, a value, and a timestamp. Producers write data to topics and consumers read from topics.
Kubernetes
Service Mesh with Istio
As per description on Istio website, Service mesh is used to describe the network of microservices that make up applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often has more complex operational requirements, like A/B testing, canary rollouts, rate limiting, access control, and end-to-end authentication.
Kubernetes Concepts
Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications, as described at Kubernetes website.
Working with K3s using Multipass
One of the easiest way to try out k3s is by using multipass, a tool that allows you to spin up Ubuntu VMs in seconds on MAC, Linux or Windows machine.
Distributed Computing
Service Mesh with Istio
As per description on Istio website, Service mesh is used to describe the network of microservices that make up applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often has more complex operational requirements, like A/B testing, canary rollouts, rate limiting, access control, and end-to-end authentication.
Running Memcached on Ubuntu
Memcached is an in-memory key-value, distributed memory object caching system, generic in nature, but originally intended for use in speeding up dynamic web applications by alleviating database load.
Spark
Apache Spark - Stream Processing
Initially big data started with collecting huge volume of data and processing it in smaller and regular batches using distributed computing frameworks such as Apache Spark. Changing business requirements needed to produce results within minutes or even in seconds.
Apache Spark
Initially software applications were small which can be deployed on a single computer. Over a period of time, data volume processed by these application grew in size. Hence, the requirement for storage and computing power grew. These requirements were fulfilled by rapid advancements in storage and compute hardware by having larger disks and faster CPUs. This way of scaling is termed as vertical scaling, which soon became costlier when applications started being consumed over Internet.
AWS
AWS EC2
EC2 is one of the most popular AWS offering. It provides the capabilities of:
- Renting Virtual Machines (EC2)
- Storing data on the virtual drives (EBS)
- Distributing load across machine (ELB)
- Scaling a service using auto-scaling group (ASG)
C
CGO - Bridge between C and Go
cgo allows Go and C programs to interoperate. This is very useful feature which lets C existing libraries to be called by Go. On the other hand Go libraries can also be called by C code.
Cloud
Cloud Setup on Macbook
Setting up cloud development machine can be challenging, especially for new comers. Here is what I did to setup various components on Macbook.
Redis
Redis Client In Golang
Remote dictionary service (redis) is an in-memory key-value data structure store used as a cache/database. Generally, it used as a cache in front of another persistent database to speed up application performance.
Storm
Apache Storm
Apache Storm is a free and open source distributed realtime computation system. Apache Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.