How do you organize your stuff? Are you one of those people who keep everything?

I’ve always taken pride in being minimalist, keeping my stuff to a minimum and I firmly believe it makes my life easier. I am an organization freak and I need to live in an organized space, where I know where my stuff is and where I can find them. Being minimalist certainly helps me a lot. I don’t have to dispend much effort organizing stuff, simply because I don’t have much stuff. I apply this to my hard drives, too.

But I’ve recently come to have a regret about this: I didn’t keep software I wrote.

When I was 16 years old, I bought a book on PHP. I spent the next days reading it, through night and day. Everything there was to learn in that book, I learned. Took me about two weeks, if I recall correctly. I already bought that book with a purpose. I always wanted to have a blog (I always liked writing, indeed, and that must be because I always loved reading), and I didn’t want to use Blogger and other private platforms, for, in my opinion, they were very poor. I could do better than that, I thought.

Then, the same day I finished reading that book, I started writing my blog manager system. After two months, I had a fully functional system. It could handle templates (simply reading the template and doing some string manipulation to generate the final output), managed posts, and it was multi-user. Yes, you could actually have a blog with multiple authors, and even define different privileges based on roles you could create. It could do a lot of stuff. I was very proud, and still am. How many people can do all that, at the age of 16, with no programming language background (other than some Pascal, I must confess) and no technical knowledge of how computers really worked?

OK, it was not beautiful. It was almost monolithical (everything on a couple of files, and within a few functions), no object-orientation, no security (like input sanitization, I didn’t even know what that was). No software engineering. I just sat on my computer, day after day, and coded, coded, coded. But then again, how many people could do all that under my circumstances?

Turns out years passed, I changed computers, hard drives were gone, and I was never much of a backup-doer. So, eventually, I lost it, just like I used to lose everything when I changed computers and I just started fresh. Don’t blame me. When I was 16 years old, there was no such thing as cloud storage, or GitHub, or distributed version control at all. I didn’t even use version control to write my CMS. External hard drives? An expensive novelty. Even CD burners were expensive.

And I lost a few other software projects as well. None as big, though.

The point is, now I have nothing but my word about it. I would like to browse through that code again, and see how much changed, how much I evolved… It would be an interesting thing to see right there.

Locking is important in multi-processor or pre-emptable systems. Data cannot be written and read at the same time by different tasks, or bizarre, random errors will occur, which may even damage the whole system.

Atomic Operations

These are operations that cannot be interrupted by any means. They are not technically locking, but they don’t need locking because they work as if they were locked already.

To use these operations, you declare a atomic_t variable. Then, you pass that variable to proper functions that perform simple arithmetic or bit-by-bit atomic operations: some functions can add an integer value atomically, others can increment the value, others can set, unset or flip a single bit, and many other functions exist.

Spinlocks

Spinlocks are the most simple and cost effective locking mechanism we have available.

If the task attempts to acquire the lock and the lock is available, the lock is acquired and the task can proceed. If the lock is unavailable, the task keeps trying to acquire the lock until it succeeds. That’s the spin part.

There are many pitfalls and consequences to the use if this lock: the lock must be held for the minimum time frame possible, because any time spinning is time lost. If your process takes a lock to protect a device from being used, and that device issues an interrupt, the interrupt handler will try to acquire the lock but it will be unavailable. The processor will spin forever. The code should not sleep with a lock acquired, as it may cause another thread to try to acquire a lock acquired by code that is not even running, causing it to spin until it loses the processor (more time is lost).

More complex methods can entirely disable interrupts, or just software interrupts. Sleep can be avoided by being careful about function calls.

There’s also a non-blocking version of this lock, using specific functions for that.

Reader/Writer Spinlocks

Usually, data be can simultaneously read by many tasks concurrently, but writing can only be done by one task at a time, to avoid overwriting and generating incorrect data.

As a consequence, there is also the reader/writer spinlock, that is analogous to the standard spinlock, except that it allows any number of readers access the critical section, but writers must have exclusive access.

The reader/writer spinlock should be used only if writer access is rarely required. For writers have exclusive access, they have priority. Therefore, once a writer gets access, no reader will get it until all writers are done. Therefore, this can cause reader starvation, denying reader access for a long time.

There are also functions to disable interrupts, and the same pitfalls that apply for the standard spinlock applies here as well.

Semaphores

Semaphores are the sleeping locks. On contention, instead of spinning, they cause the task to sleep. They also allow for a certain number of tasks to acquire access to the critical section (or resource, in the contect of semaphores), instead of only one.

Semaphores count the number of resources available. That’s their initial value. When a thread attempts to acquire access to the critical section, it decrements that counter. If the new counter is zero or more, the access is granted. If the new counter is a negative value, the access is denied and the thread is sent to sleep. When a task comes out of the critical section, it increases the counter by one. Consequently, the counter can be seen as counting the number of resources available if it’s positive or the number of tasks waiting to enter the critical section if it’s a negative number. If it’s zero, means there are no resources available, but there are no tasks waiting either.

Semaphores are more expensive than spinlocks, but they can held for longer periods of time, since they sleep when access is denied.

When the initial value of the counter of a semaphore is 1, the semaphore is called a mutex (stands for mutual exclusion), because only one task can be in the critical section at any given instant.

Reader/writer Semaphores

Analogous to the reader/writer spinlocks, there are the reader/writer semaphores. There can be as many readers as resources available, but there can only be one writer at any given moment.

RCU

RCU (stands for read-copy-update) is a synchronization method that allows as many readers and writers in the critical section at any given time.

The method uses a publish-subscribe framework that keeps things straight while tasks read and concurrently modify the contents of data. Usually, there’s a list that holds the current valid or in use data structures. Every time a task creates a new data structure, it publishes it (adds it to that list). Every time a task needs to read that data structure, it uses a pointer to subscribe to the list. The pointer is pointed to the latest published data structure and can be used by that task. Therefore, even if the data structure is modified by another task, the current task will work with the one it has now until it’s done. When it subscribes again, it will receive the latest data structure.

Those accesses are monitored, and every time an outdated data structure is not being used by any task anymore, it is reclaimed.

The Big Kernel Lock

The last lock discussed is the Big Kernel Lock (BKL). It’s discussed just out of curiosity, since it’s been extinct for quite a while now, and shouldn’t be used anymore, since it’s dangerous and kills the purpose of pre-emption and concurrency of the whole system.

This lock is a global lock. Every thread that tries to acquire this lock is competing with every other thread in the whole system that tries to acquire it, not only in the process. It was introduced in the kernel 2.0 as the first kind of locking mechanism in the system, and it’s been gradually replaced by more meaningful, efficient and safe locking mechanisms (the ones discussed previously) until its extinction.

Conclusion

For each situation in which a lock is needed, there are many alternatives. Careful thinking is required to decide the best way to handle the problem, taking into consideration efficiency and security. Locking issues can severely slow down the system, completely freeze it in a deadlock, open security holes or generate completely bizarre, random errors when data is simultaneously written by diverse tasks. It’s been a topic of intense research in the computing industry and academia and there’s a lot yet to be found.

For so long, I thought I had grasped the concepts of pointers, arrays, and the subtleties in between. For so long…

Until a few days ago, after some discussions on IRC, some questions in Stack Overflow and talking to some friends, plus a lot of tests, I saw my concepts slowly changing into something that made more sense.

What I thought I knew

Long story short, I thought pointers and arrays were exactly the same thing, but had different notations. Guess what? Turns out they are not!

That’s tricky, and it’s not completely unreasonable to think otherwise. Check out the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#include <stdio.h>                                                              
 
int main(void)                                                                  
{                                                                               
        int array[5] = {0, 1, 2, 3, 4};                                         
        int *ptr;                                                               
 
        /* Array assigned to pointer. */
        ptr = array;                                                            
 
        printf("Print first element using 'array': %d\n"                        
                "Print third element using 'ptr': %d\n",                        
                array[0], ptr[2]);                                              
 
        return 0;                                                               
}

The compiler won’t issue a single warning on that. That’s perfectly standards-compliant code. And let’s face it, that can so easily trick a newbie into thinking that arrays and pointers are the same, because it looks like the name of an array is a pointer to its first element. But, if they were the same, the following code would work:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#include <stdio.h>                                                              
 
int main(void)                                                                  
{                                                                               
        int array[5] = {0, 1, 2, 3, 4};                                         
        int **p;                                                                
 
        /* p is a pointer to a pointer to int. If array were a pointer to int,  
           then this line would work, because &array would return a pointer to   
           a pointer. */
        p = &array;                                                             
 
        printf("First element of array: %d\n", **p);                            
 
        return 0;                                                               
}

But the code above won’t compile. Compiler will complain of assignment from incompatible pointer type in line 11, because it tries to assign a int (*)[5] to a int **.

And that can go unnoticed for as long as a few years (some five or six years, in my case!).

The truth (or so it seems)

What exactly is a pointer and an array? A pointer is a variable that stores a number that is a memory address. An array is a set of variables that are declared together and are placed adjacent in memory.

Upon declaration of an array, no pointer is created.

What really happens is that every time the name of an array is used with no subscript operator, it decays to be the address of its first element. And addresses can be assigned to pointers. But the variable is not a pointer in itself.

And there are formal types: type int * and int [5], for instance. First is a pointer to an int, latter is an array of five ints, and they are not interchangeable. Remember that for an array to be placed in a pointer, it first decays to an address, and addresses can be placed in pointers.

How does that impacts the code? Here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

#include <stdio.h>                                                              
#include <stdlib.h>                                                             
 
int main(void)                                                                  
{                                                                               
        int array[5] = {0, 1, 2, 3, 4};                                         
 
        int *p1; /* p1 is a pointer to an int */                                
        int **p2; /* p2 is a pointer to a pointer to an int */                  
        int *p3[5]; /* p3 is an array of 5 pointers to ints */                  
        int **p4[5]; /* p4 is an array of 5 pointers to pointers to ints */     
        int (*p5)[5]; /* p5 is a pointer to an array of 5 ints */               
        int (**p6)[5]; /* p6 is a pointer to a pointer to an array of 5 ints */ 
        int *(*p7)[5]; /* p7 is a pointer to an array of 5 pointers to ints */  
 
        /* Allocating pointers: */                                              
        /* Allocates space to the pointer that points to an int */              
        p2 = malloc(sizeof(int *));                                    
        /* Allocates space to the pointer that points to an int */              
        p4[0] = malloc(sizeof(int *));                                 
        /* Allocates space to the pointer that points to an array of ints */    
        p6 = malloc(sizeof(int (*)[5]));                          
        /* Allocates space to an array of five pointers to ints */              
        p7 = malloc(sizeof(int *[5]));                            
 
        /* The possible combinations: */                                        
 
        p1 = array; /* array decayed to an address to its first element */      
        *p2 = array; /* The contents of p2 (a pointer to an int) receives array 
                        (decayed to an address to its first element) */         
        p3[0] = array; /* First element of p3 is a pointer to an int */         
        *p4[0] = array; /* The first element of the array pointed by p4         
                           receives the address of the first element of the     
                           array of ints */                                     
        p5 = &array; /* p5 is a pointer to an array of 5 ints, therefore must   
                        receive &array */                                       
        *p6 = &array; /* p6 receives a pointer to an array of 5 ints */         
        (*p7)[0] = array; /* p7 is dereferenced to be the array of pointers to  
                             int. The first element of that array is then       
                             accessed by the subscript operator and receives    
                             array, that decayed to be the address of its first 
                             element */                                         
 
        /* Printing elements of arrays, using the pointers: */
        printf("Printing the first element, using:\n");                         
        printf("\tp1: %d\n", p1[0]);                                            
        printf("\tp2: %d\n", (*p2)[0]);                                         
        printf("\tp3: %d\n", p3[0][0]);                                         
        printf("\tp4: %d\n", (*p4[0])[0]);                                      
        printf("\tp5: %d\n", (*p5)[0]);                                         
        printf("\tp6: %d\n", (**p6)[0]);                                        
        printf("\tp7: %d\n", (*p7)[0][0]);                                      
        printf("All output should be zero.\n");                                 
 
        /* Freeing allocated pointers: */                                       
        free(p7);                                                               
        free(p6);                                                               
        free(p4[0]);                                                            
        free(p2);                                                               
 
        return 0;                                                               
}

A code that is supposed to work the same way, but does not take into account the subtle difference between a pointer and an array, won’t even compile.

One of the most powerful features of Unix systems is that almost everything has full support for command line interface and text.

You almost never depends on the mouse to do anything, although you almost always have the option to use it.

That’s a powerful feature, because text is the simplest kind of data to parse and process, and so everything you can do with text only, you can easily automate.

One interesting thing you can with that is completely automate the installing process of your system.

When you install a system for the first time, it takes several hours (maybe even days) to get it working exactly the way you want, assuming you don’t want the default settings on everything. You have to install applications. You have to configure your desktop overall settings, like appearances and behaviors. And you have to configure all applications you use.

In Unix systems, since all configuration is stored in text files (usually as hidden files or inside hidden folders in your home folder), you can change all your options by simply editing those files. And more powerfully yet, you can backup those files and restore all your options simply overwriting the default ones, copying those files inside your home directory.

And since Unix is so easily configurable, you can create a simple script and completely automate the process of setting up your whole system to your taste from a fresh install.

Of course, that’s only possible because installing applications is automatic as well: no need to have someone in front of the computer pressing the next button multiple times.

So that’s what I do: I have a file that lists all applications I have installed in my computer; I have all my configuration files backed up; I wrote a simple script that installs everything with a command; and I have a remote repository for that.

Every time I want to change some setting, I change those files and commit them to the remote repository. If I ever need to recover my settings from some failure, or want to reinstall the whole system fresh, all I need to do is pull the repository to my local disk and run the install script. When I reboot, it will be as if the system was never reinstalled at all. Even if I change my hardware.

I never said it’s not possible to automate that in the next-button-lover operating system, but I bet it’s way harder. You’ll have to, at least, pay someone to press the next button for you all day long!

Pair programming, and agile techniques in general, have become quite popular. Everyone’s doing it.

There are sophisticated frameworks to ease pair programming, most of them integrated with IDEs. They make remote pair programming rather easy. But there’s a rather simple and effective way of implementing remote pair programming, using just tools that should be easily available in every Linux distribution.

Pair programming requires a few things:

• a partner;
• an effective way to communicate (preferably voice);
• and a synchronized screen.

The first requirement is left up to you!

The second requirement is easy too. Any voice communication software will do, I use Skype.

The third one can be implemented using SSH and tmux. tmux is a terminal multiplexer that supports a client/server model. It means two or more terminals can connect to the same tmux session and share that session, with all terminals acting as if they were one, including taking input from all. It also means any terminal can detach from a session and resume it later, while the session keeps running in the background if all terminals are detached.

Here’s how you can do it.

First, you need your partner to SSH into your machine.

Then, you need to start tmux. Set a session inside your /tmp directory using this:

$ tmux -S /tmp/session_name

Then, tell your partner to start tmux telling it to use that same session with this line:

$ tmux -S /tmp/session_name attach

Of course, you need to make sure your partner has permissions to read and write to that session file as well.

And that’s it, just fire up your favorite text editor in terminal mode and code!

I’ve used this setting many times and works great. And it demands very little hardware and bandwidth, since everything being synchronized is just text.