Tag Archive

golang

Golang Notes

Golang is an open-source, compiled, and statically typed programming language designed by Google. It is built to be simple, high-performing, readable, and efficient.

Redis Client In Golang

Remote dictionary service (redis) is an in-memory key-value data structure store used as a cache/database. Generally, it used as a cache in front of another persistent database to speed up application performance.

CGO - How to use C shared library code

I was working with CGO code and realized the need of keeping the C code in a shared library and using it from Go code. Here is a brief run down of how to go about it.

CGO - Bridge between C and Go

cgo allows Go and C programs to interoperate. This is very useful feature which lets C existing libraries to be called by Go. On the other hand Go libraries can also be called by C code.

GRPC Tutorial

I will be using Golang in this post. Let’s install the software needed. There are equivalent commands to try on Raspberry-PI.

Golang Testing

Testing a.k.a. automated testing, is the practice of writing small programs that check that the code under test behaves as expected for certain inputs, which are either carefully chosen to exercise certain features or randomised to ensure broad coverage.

Back to Top ↑

linux

Linux Virtualization

Virtualization solutions allow multiple operating systems and applications to run in independent partitions on a single computer. Using virtualization capabilities, one physical computer system can function as multiple “virtual” systems.

CGO - How to use C shared library code

I was working with CGO code and realized the need of keeping the C code in a shared library and using it from Go code. Here is a brief run down of how to go about it.

Extended Berkeley Packet Filter (eBPF)

eBPF stands for extended Berkeley Packet Filter but officially referred to as BPF. BPF was intially developed for high performance packet capture in 1992. In recent years (2012 - 2014), BPF is rewritten to a general purpose virtual machine that can be used for many things like networking, monitoring, security and performance analysis.

Netfilter and IPTables

iptables is a management tool for the firewall software netfilter in the Linux kernel. iptables is located in the user space while netfilter is located in the kernel space, where functionalities of firewalling, network address translation (NAT), packet content modification and packet filtering is implemented. Both together are commonly referred to just iptables.

Sockets

Sockets are the preferred way of network communication. They are provided by Operating Systems as socket API, which are based on the principles of reading and writing files.

Back to Top ↑

Kafka

Kafka Producer and Consumer In Golang

Apache Kafka is a distributed publish-subscribe messaging system that is designed to be fast, scalable, and durable. Kafka stores streams of records (messages) in topics. Each record consists of a key, a value, and a timestamp. Producers write data to topics and consumers read from topics.

Patterns for Composite Containers

Building an application from modular containers means thinking about symbiotic groups of containers that cooperate to provide a service, not one container per service. In Kubernetes, the embodiment of this modular container service is a Pod. A Pod is a group of containers that share resources like file systems, kernel namespaces and an IP address. The Pod is the atomic unit of scheduling in a Kubernetes cluster, precisely because the symbiotic nature of the containers in the Pod require that they be co-scheduled onto the same machine, and the only way to reliably achieve this is by making container groups atomic scheduling units.

Kafka Connect and Streams APIs

Here is a typical architecture having Sources, Sinks, Connect Cluster, Kafka Cluster and Kafka Streams Applications.

Kafka Basics

Apache Kafka is a distributed publish-subscribe messaging system that is designed to be fast, scalable, and durable. Kafka stores streams of records (messages) in topics. Each record consists of a key, a value, and a timestamp. Producers write data to topics and consumers read from topics.

Back to Top ↑

distributed computing

Redis Client In Golang

Remote dictionary service (redis) is an in-memory key-value data structure store used as a cache/database. Generally, it used as a cache in front of another persistent database to speed up application performance.

Service Mesh with Istio

As per description on Istio website, Service mesh is used to describe the network of microservices that make up applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often has more complex operational requirements, like A/B testing, canary rollouts, rate limiting, access control, and end-to-end authentication.

Running Memcached on Ubuntu

Memcached is an in-memory key-value, distributed memory object caching system, generic in nature, but originally intended for use in speeding up dynamic web applications by alleviating database load.

Back to Top ↑

kubernetes

Service Mesh with Istio

As per description on Istio website, Service mesh is used to describe the network of microservices that make up applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often has more complex operational requirements, like A/B testing, canary rollouts, rate limiting, access control, and end-to-end authentication.

Kubernetes Concepts

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications, as described at Kubernetes website.

Back to Top ↑

Kafka Connect

Patterns for Composite Containers

Building an application from modular containers means thinking about symbiotic groups of containers that cooperate to provide a service, not one container per service. In Kubernetes, the embodiment of this modular container service is a Pod. A Pod is a group of containers that share resources like file systems, kernel namespaces and an IP address. The Pod is the atomic unit of scheduling in a Kubernetes cluster, precisely because the symbiotic nature of the containers in the Pod require that they be co-scheduled onto the same machine, and the only way to reliably achieve this is by making container groups atomic scheduling units.

Kafka Connect and Streams APIs

Here is a typical architecture having Sources, Sinks, Connect Cluster, Kafka Cluster and Kafka Streams Applications.

Back to Top ↑

Kafka Streams

Patterns for Composite Containers

Building an application from modular containers means thinking about symbiotic groups of containers that cooperate to provide a service, not one container per service. In Kubernetes, the embodiment of this modular container service is a Pod. A Pod is a group of containers that share resources like file systems, kernel namespaces and an IP address. The Pod is the atomic unit of scheduling in a Kubernetes cluster, precisely because the symbiotic nature of the containers in the Pod require that they be co-scheduled onto the same machine, and the only way to reliably achieve this is by making container groups atomic scheduling units.

Kafka Connect and Streams APIs

Here is a typical architecture having Sources, Sinks, Connect Cluster, Kafka Cluster and Kafka Streams Applications.

Back to Top ↑

Spark

Apache Spark - Stream Processing

Initially big data started with collecting huge volume of data and processing it in smaller and regular batches using distributed computing frameworks such as Apache Spark. Changing business requirements needed to produce results within minutes or even in seconds.

Apache Spark

Initially software applications were small which can be deployed on a single computer. Over a period of time, data volume processed by these application grew in size. Hence, the requirement for storage and computing power grew. These requirements were fulfilled by rapid advancements in storage and compute hardware by having larger disks and faster CPUs. This way of scaling is termed as vertical scaling, which soon became costlier when applications started being consumed over Internet.

Back to Top ↑

C

CGO - Bridge between C and Go

cgo allows Go and C programs to interoperate. This is very useful feature which lets C existing libraries to be called by Go. On the other hand Go libraries can also be called by C code.

Back to Top ↑

Storm

Apache Storm

Apache Storm is a free and open source distributed realtime computation system. Apache Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

Back to Top ↑

aws

AWS EC2

EC2 is one of the most popular AWS offering. It provides the capabilities of:

  • Renting Virtual Machines (EC2)
  • Storing data on the virtual drives (EBS)
  • Distributing load across machine (ELB)
  • Scaling a service using auto-scaling group (ASG)
Back to Top ↑

cicd

Back to Top ↑

cloud

Cloud Setup on Macbook

Setting up cloud development machine can be challenging, especially for new comers. Here is what I did to setup various components on Macbook.

Back to Top ↑

k3s

Back to Top ↑

memcached

Running Memcached on Ubuntu

Memcached is an in-memory key-value, distributed memory object caching system, generic in nature, but originally intended for use in speeding up dynamic web applications by alleviating database load.

Back to Top ↑

redis

Redis Client In Golang

Remote dictionary service (redis) is an in-memory key-value data structure store used as a cache/database. Generally, it used as a cache in front of another persistent database to speed up application performance.

Back to Top ↑

virtualization

Linux Virtualization

Virtualization solutions allow multiple operating systems and applications to run in independent partitions on a single computer. Using virtualization capabilities, one physical computer system can function as multiple “virtual” systems.

Back to Top ↑