How the quest to prevent time from running out has led me to all corners of the Linux kernel

The y2038 problem in Linux is about the time data type representation. Because of the way time is represented in Linux, a signed 32 bit number can not support times beyond January 2038. The solution is to use 64 bit timestamps.

I came into the problem as Arnd Bergmann’s intern for Outreachy. Outreachy is a benevolent program which helps new programmers to get into open source development. The mentors for the kernel projects are usually experienced kernel developers.

I chose the y2038 problem because it would let me touch all the subsystems in the kernel and it actually did more than that. It also involves user space, C library, POSIX and C standards. And I found that the problem is really about interfaces between layers.

This article is about how solving one problem does not usually involve just solving that one problem in the kernel: the complexity of interrelated things in the kernel (there is always 1 more cleanup that is needed before the change) and the interactions with the community as a newcomer.

One of the problems we tackled was the virtual file system.

VFS is a filesystem abstraction layer. So even if some of the filesystems like the ext4 were able to represent timestamps beyond the year 2038 on a 32 bit system, they cannot do so without the VFS layer supporting it.

This change to VFS was one of the patch series that took the longest to get consensus and get merged in.

Problem Description: The in-kernel representation of inode timestamps was in struct timespec which is not y2038 safe. Change the representation to struct timespec64 which is y2038 safe.

The problem is an example of how the y2038 problem touched many sub systems in the kernel.

The first version of the series was posted by Arnd in 2014. There were a few open issues and there was some feedback about adding timestamp range checking at that time.

I posted the first request for comments for this in Jan of 2016. The RFC was really about asking if there was any opposition to the approach. This was  not a typical RFC for the kernel community. The series cover letter said we could change it this way, and provided a few example changes of how this would be done. There was some confusion here as to what we were trying to get across in the series.

So again I posted a series (actually 3) for solving the problem in 3 separate ways. This was a pared down version of the earlier series addressing only the core issue. This was also atypical. Thomas suggested that he slightly preferred using one of the approaches to solving the problem.

Now, we had all the patches done this way. But, we had to get rid of some old time interfaces before we could actually do the change. When I posted a series of this, Linus did not like one of the interfaces (current_fs_time(sb)) since it took the superblock as an argument, to access timestamp granularity. But the timestamps are really a feature of the inode, not the super block. So we ended up getting rid of this API.

Now, the original series had to be redone again. But, it seemed like doing a flag day patch was a brute force approach to the problem. So we ended up doing just that.  We even went a step further to do it with a coccinelle script. This changed over 80 files. The challenge here was to make the changes rudimentary to avoid regressions. We finally ended up getting the patches merged in June 2018. We have not yet heard of any regressions the change introduced.

So by the end of this whole exercise, we had gotten rid of three in-kernel APIs, rearranged some of the filesystem timestamp handling, handled print formats to support larger timestamps, analyzed 32 bit architecture object dumps and rewritten at least five versions of the series from scratch. And, this was just one of the problems we solved for the kernel.

Y2038 has been one of my favorite projects.

ELC 2016 Trip

I had already recently been to one conference, Linaro Connect 2015 in Bangkok. I hadn't really been able to convince myself that would be useful. But it turned out that it was quite helpful to meet in person with my project mentor. We learned more about each other, which was helpful to continue our work on the project even after the internship.

Arnd recommended the Embedded Linux Conference, because it is attended by a more general sampling of kernel developers. Linaro is focused on the ARM ecosystem.

I later figured that if I spoke to people at the conference about the patches, it would be helpful. This was necessary because the patches I submitted were still under discussion on the linux kernel mailing list. The problem of transitioning VFS to 64 bit timestamps involves many other tiny problems. So it was getting difficult to convey the big picture effectively to everybody. Longer patch series usually tend to be ignored. I shared this with Arnd, and he agreed it would be a good idea, so we made a list of people that would be interested in the project.

I still had some funds on offer from the Outreachy internship, so in the end I decided it would be a useful expedition to visit San Diego.

Each evening, I made a plan for the next day to choose what sessions I might attend the next day.

It was interesting to observe different kinds of talks. Some were very popular, while others seem to have very narrow user base.

I tried to attend talks which are aligned with my work and interests. Some of the talks that I attended were about memory barriers, kernel debugging, and static analysis.

Arnd introduced me to various kernel maintainers, linux developers, and former colleagues. One person pointed out that people would want to talk with you and learn your name so that they could look out for your patches. So it helps wearing the name tag :).

We met tglx and did some code review, then discussed a plan for how to upstream the supporting VFS patches. We also gave Linus a brief update about the status of the problem.

Arnd and I also did more code reviews to streamline the patches.

So it was a good trip because we made progress on the project.

I also met many other interesting people. Lunches were fun, too. I went with Arnd's family one day. I also ended up going to lunch with Karen Sandler and my old lead, and we were discussing various interesting topics.

All in all it was a wonderful and productive trip.

Outreachy application process

Number of patches submitted: 5

Choosing the project

I first started by picking the project and mentors.

I had a specific goal coming to work on the kernel. I have previously worked on many boot loaders at my previous job and was now looking for more of a challenge. I had previously chosen boot loader because I did not want to get lost in the software, but actually wanted to learn how one small OS could keep the hardware working. The kernel was a good option because of the challenges involved with multiple architectures, bigger operating system, and working with the community itself.

I read about mentors and I thought Arnd was a good fit for me. The project was also a good fit as the scope of it was in line with my goals. I did not want to work on one portion of the system, or on drivers. I wanted to touch as many subsystems as possible.

Picking the patches

As the application process suggests, I wanted to be considered for the internship. I wanted to submit a patch that would be accepted and not controversial so my application could be considered. I wanted to get it out of the way. This was also because I feared that if I picked a complicated patch to begin with, I might not be able to complete it in time. This also let me test my email set up, right tree configuration etc.

The 2 patches fixed a couple of checkpatch errors. I did not get many suggestions on these, except that Julia had a style preference for macros.

Connect with the mentor

I have always been quite thorough with my work. So I would figure out how to test after design. I wrote to Arnd about it and he suggested that the community makes small patches, so a compile test and rigorous review will suffice most of the times.

Next I wrote to Arnd for some subtasks. I believed I needed a dedicated effort so that I could start diving deeper into the kernel. Arnd suggested that I start with easy and work my way up to tricky problems on his task list.

Introductory problem

I first started with learning about different timers. I read the timer docs from the Documentation directory and then the paper by John Stultz. It was easy to for me to understand timers and the actual hardware blocks used because of my experience. I had written a driver to use timers to support context switching in my boot loader which used round robin scheduling. This was also my favorite driver as it showed how the basic blocks work: scheduling, timers, interrupts, context switching etc.

Next I fixed a staging driver. But, while Arnd and I were arguing over why I had chosen to do the patch this way, Arnd discovered that the whole driver might not be required any longer. Greg suggested I delete it and so much for my first patch! But Arnd was gracious enough to let me do the patch to do the removal.

Medium problem

Next I chose a medium problem: usbtest ioctl

This was quite fun as I got to learn that this was implemented as a USB driver. The higher level framework was doing the conversions for compat mode. I again read a bit about ioctls, looked at header files, figured out what uapi files are, also did some research on CONFIG_COMPAT ioctl.

I got a little pedantic and introduced a new ioctl. I said the ioctl interface was never implemented correctly. Arnd gently steered me towards just fixing the data types. I ended up adding ioctls for 2 different input structures of different sizes.

Tricky problem

Next I chose MD problem, which was called a tricky problem.

This needed a little understanding of the MD driver and RAID.

I interacted with Neil, and Arnd had already started a discussion on the right approach.

Conclusion

These were the only patches I submitted through the application period, and my last one was merged after the application period was closed.

I was looking for someone who could see that these patches were different and could see my potential. I did not want to do more clean up patches. This approach worked for me.