Redis Modules: an introduction to the API

The modules documentation is composed of the following files:

  • INTRO.md (this file). An overview about Redis Modules system and API. It’s a good idea to start your reading here.
  • API.md is generated from module.c top comments of RedisMoule functions. It is a good reference in order to understand how each function works.
  • TYPES.md covers the implementation of native data types into modules.
  • BLOCK.md shows how to write blocking commands that will not reply immediately, but will block the client, without blocking the Redis server, and will provide a reply whenever will be possible.

Redis modules make possible to extend Redis functionality using external modules, implementing new Redis commands at a speed and with features similar to what can be done inside the core itself.

Redis modules are dynamic libraries, that can be loaded into Redis at startup or using the MODULE LOAD command. Redis exports a C API, in the form of a single C header file called redismodule.h. Modules are meant to be written in C, however it will be possible to use C++ or other languages that have C binding functionalities.

Modules are designed in order to be loaded into different versions of Redis, so a given module does not need to be designed, or recompiled, in order to run with a specific version of Redis. For this reason, the module will register to the Redis core using a specific API version. The current API version is “1”.

This document is about an alpha version of Redis modules. API, functionalities and other details may change in the future.

https://github.com/antirez/redis/blob/unstable/src/modules/INTRO.md

Python by the C side

All the world is legacy code, and there is always another, lower layer to peel away. These realities cause developers around the world to go on regular pilgrimage, from the terra firma of Python to the coasts of C. From zlib to SQLite to OpenSSL, whether pursuing speed, efficiency, or features, the waters are powerful, and often choppy. The good news is, when you’re writing Python, C interactions can be a day at the beach.

https://www.paypal-engineering.com/2016/09/22/python-by-the-c-side/

How to Read and Write Other Process Memory

I recently put together a little game memory cheat tool called MemDig. It can find the address of a particular game value (score, lives, gold, etc.) after being given that value at different points in time. With the address, it can then modify that value to whatever is desired.

I’ve been using tools like this going back 20 years, but I never tried to write one myself until now. There are many memory cheat tools to pick from these days, the most prominent being Cheat Engine. These tools use the platform’s debugging API, so of course any good debugger could do the same thing, though a debugger won’t be specialized appropriately (e.g. locating the particular address and locking its value).

My motivation was bypassing an in-app purchase in a single player Windows game. I wanted to convince the game I had made the purchase when, in fact, I hadn’t. Once I had it working successfully, I ported MemDig to Linux since I thought it would be interesting to compare. I’ll start with Windows for this article.

http://nullprogram.com/blog/2016/09/03/

MemDig: a memory cheat tool

MemDig allows the user to manipulate the memory of another process, primary for the purposes of cheating. There have been many tools like this before, but this one is a scriptable command line program.

There are a number of commands available from the program’s command prompt. The “help” command provides a list with documentation. MemDig commands can also be supplied as command line arguments to the program itself, by prefixing them with one or two dashes.

All commands can be shortened so long as they remain unambiguous, similar to gdb. For example, “attach” can be written as “a” or “att”.

The current set of commands is quite meager, though it can operate on integers and floats of any size. The command set will grow as more power is needed.

https://github.com/skeeto/memdig

An Introduction to Crystal: Fast as C, Slick as Ruby

So I’m gonna ask you a question:

To be honest, I’ve always dreamed of something like that and wondered why it didn’t exist. Then I found Crystal. I still remember it clearly: It was July 2015, I was reading /r/programming, and I saw something like “Crystal: Fast as C, Slick as Ruby.”

https://blog.codeship.com/an-introduction-to-crystal-fast-as-c-slick-as-ruby/

Appending to a File from Multiple Processes

Suppose you have multiple processes appending output to the same file without explicit synchronization. These processes might be working in parallel on different parts of the same problem, or these might be threads blocked individually reading different external inputs. There are two concerns that come into play:

1) The append must be atomic such that it doesn’t clobber previous appends by other threads and processes. For example, suppose a write requires two separate operations: first moving the file pointer to the end of the file, then performing the write. There would be a race condition should another process or thread intervene in between with its own write.

2) The output will be interleaved. The primary solution is to design the data format as atomic records, where the ordering of records is unimportant — like rows in a relational database. This could be as simple as a text file with each line as a record. The concern is then ensuring records are written atomically.

http://nullprogram.com/blog/2016/08/03/

Building a BitTorrent client from scratch in C#

BitTorrent is a protocol for peer-to-peer file sharing. It allows users to directly share files with each other across the internet without any central server acting as a middleman.

In order to do this, the files are divided up into small regular-sized pieces. Each client or peer in the network can then either request a piece (if it is missing it) or send a piece (if another peer requests it). Peers can send and receive pieces simultaneously from multiple other peers until all peers have the complete file. A peer is called a seeder if it has pieces available to send out and a leecher if they are still requesting pieces.

The lack of a central server means that there is bandwidth costs of sharing content is reduced for the originator. Initially there will be a single seeder, however once other peers obtain the files they become seeders too. The protocol tends to favour more popular content. The more peers that want a file, the more peers there will be that have the file to share. Supply scales with demand. In this regard, it is also a more resilient method as the network becomes resistant to a system failure and does not have any single point of failure once there are multiple seeders.

Unpopular content can be difficult or slow to download if there are only a handful of seeders. Small files can be slower to download than from a traditional server as there is an certain amount of time overhead finding peers. The lack of a central server can also lead to a situation where all of the peers in the network are almost complete but all missing the same piece (although this should be rare due to the algorithms used to select pieces to request).

https://cheatdeath.github.io/research-bittorrent-doc/

Bit Twiddling Hacks

Individually, the code snippets here are in the public domain (unless otherwise noted) — feel free to use them however you please. The aggregate collection and descriptions are © 1997-2005 Sean Eron Anderson. The code and descriptions are distributed in the hope that they will be useful, but WITHOUT ANY WARRANTY and without even the implied warranty of merchantability or fitness for a particular purpose. As of May 5, 2005, all the code has been tested thoroughly. Thousands of people have read it. Moreover, Professor Randal Bryant, the Dean of Computer Science at Carnegie Mellon University, has personally tested almost everything with his Uclid code verification system. What he hasn’t tested, I have checked against all possible inputs on a 32-bit machine. To the first person to inform me of a legitimate bug in the code, I’ll pay a bounty of US$10 (by check or Paypal). If directed to a charity, I’ll pay US$20.

https://graphics.stanford.edu/~seander/bithacks.html

Understanding glibc malloc

I always got fascinated by heap memory. Questions such as

How heap memory is obtained from kernel?
How efficiently memory is managed?
Is it managed by kernel or by library or by application itself?
Can heap memory be exploited?

were in my mind for quite some time. But only recently I got time to understand about it. So here I would like to share my fascination turned knowledge!! Out there in the wild, many memory allocators are available:

  • dlmalloc – General purpose allocator
  • ptmalloc2 – glibc
  • jemalloc – FreeBSD and Firefox
  • tcmalloc – Google
  • libumem – Solaris

https://sploitfun.wordpress.com/2015/02/10/understanding-glibc-malloc/