Problems worthy of attack prove their worth by hitting back. —Piet Hein

Friday 27 April 2018

Pastures new-ish

After nine and a half years, today is my last day at Cloudera. It’s difficult to write those words as so much of my life has been bound up with this company. On the day I started, I didn’t meet my co-workers as I was living several thousand miles away in a barn in Wales. (The others were in a borrowed meeting room in San Mateo.) As I leave I am still in a barn in Wales (different barn though), but a lot has happened in the intervening period.

On the personal side, my family and I lived in San Francisco during the early formative years of Cloudera, a time we will always treasure for the lifelong friendships we made.

On the professional side, it is no exaggeration to say that working at Cloudera has been the highlight of my career. I already knew that Hadoop was pretty special when I joined (I may have been biased as I was writing a book on it), but I had no idea how it would transform the industry and how it would be used in every sector you could imagine.

To all of you I have worked with over the last decade—at Apache, Cloudera and elsewhere, on many projects—I consider myself to be incredibly fortunate to have had the opportunity to work with you. Thank you.

So what’s next for me?

Jim Waldo, who worked on distributed systems at Sun, once said that he alternated six month periods between the lab and the outside world: in the lab he and his team built systems software, and in the outside world he saw how people used the system he was building. Doing so gave him valuable feedback on the system design, even though it was time away from being able to build the system.

In some ways this is another way of framing the explore/exploit tradeoff, where you decide between exploring new technological ground—building a new system—and exploiting that system to solve particular problems you are interested in, which is why you built the system in the first place. (Of course, this framing is oversimplified, since there are many people working on both parts simultaneously. It’s a useful way of thinking about things as an individual actor though.)

For the past few years I have been working on a few open source biology and healthcare projects (like GATK, Hail, and OHDSI). I think that the problems in biology are big enough and messy enough that new systems will need to be built. We can’t stop exploring the technological ground since the sheer amount of data will overwhelm even the best of today’s cutting-edge technology. (I like to cite the paper Big Data: Astronomical or Genomical? here for some concrete numbers.)

Having said that, there is still a lot of mileage left in our current crop of tools—which include Spark, TensorFlow, Jupyter, and the cloud. And this is what I am going to do: continue the work to apply tools like these to more bio projects, only now working as a freelancer. I plan to write more about what I’m up to on this blog, so please follow along.

Cloudera Inbox Zero for the first time ever!

Sunday 15 April 2018

Type 1 Diabetes

On 16 February this year I was diagnosed with Type 1 Diabetes.

I had been feeling under the weather - a bit weak, but also persistently thirsty and hungry. I would gulp down a large glass of water in one go with ease (a bit like I did when I was 10 years old after running around outside for hours on a hot day), and I would do this several times every day. I would eat a sandwich after dinner, even after having seconds. Strangely, I was losing weight despite eating a lot. And I found it hard to complete my usual morning run, and when I did manage it, it was noticeably slower than normal.

In retrospect these are all classic symptoms of type 1 diabetes: weight loss, increased thirst and hunger. My body was not producing enough insulin, which is needed to use the glucose in my blood. The weight loss occurred because my body was using fat reserves for energy. My heightened thirst was my body’s way of trying to flush the excess glucose from my system by getting me to urinate more.

So I went to see the doctor and she arranged a blood test, which I had the next day. That was at lunchtime on Friday, and the nurse who took my blood said it would be a week or two before the results came back. So I was surprised when the phone rang at 6pm, and the doctor’s receptionist asked me to come in. On Friday evening? Weren’t they closed then?

A few minutes later I went in, and she said that my blood glucose level was over 30, and that normally it would be 7. “Tom, you have diabetes.” I didn’t know what to say. I remember asking what the blood glucose level was measured in (millimoles per litre). (I’m always impressing on my daughters the importance of units in science.)

I also asked how they knew it was Type 1. Mainly from the symptoms - I'm quite skinny! Most people with diabetes have Type 2, which is characterised by insulin resistance: the body is still producing insulin, but it can’t use it as effectively. Less than 10% of diabetes sufferers have Type 1, and while most of those are children, adults can get the disease too.

The doctor sent me to the hospital straightaway so they could check if my body was coping with the high sugar levels. Left untreated, high blood glucose levels can lead to a dangerous condition called ketoacidosis.

I rang Eliane to tell her, and we both had a bit of a wobble. She drove over to take me to the hospital. We went to the Emergency Admissions Unit, where they would check my ketone levels (which, thankfully, were normal) and give me a shot of insulin. After that I could go home - there was no need to stay overnight, but they asked me to go back the next day and Sunday to be given insulin again.

The next day, Monday, I saw the Diabetic Specialist Nurse (DSN) who gave me a blood glucose monitor and insulin pens so I could manage the condition myself. She explained my new routine: before every meal and before bed I have to check my blood glucose level and inject insulin. The bedtime insulin is a slow-acting background insulin that lasts for almost a whole day, whereas the mealtime insulin is fast-acting and is meant to compensate for the blood glucose level rise caused by the carbohydrates in the meal. The idea is that you count the number of carbohydrates in the meal you are about to eat, then calculate the number of units of insulin that are needed to cover it.

It doesn’t sound like much, but I hadn’t really paid much attention to the nutritional composition of meals before. And nor had Eliane. We eat healthily, and mainly cook from scratch, but having to analyse each meal was a big change. In the first few weeks it felt like we were spending all our time analysing recipes. It gets easier, but it’s still time consuming.

The goal with diabetes is to keep the blood glucose level between 4 and 7 mmol/l. A person without diabetes has a pancreas that does this for them. Unfortunately, my pancreas has stopped performing this role, hence the need for injected insulin. There are two things to avoid: hyperglycaemia, which is when the blood glucose level is too high, and hypoglycaemia when it is too low.

Broadly speaking, hyperglycaemia has long term effects (such as cardiovascular problems), while hypoglycaemia needs to be treated immediately since its more severe form can require hospitalisation. Normally though, treating a mild “hypo” involves eating something with very fast acting sugar in it (like glucose tablets) and waiting 15 minutes for the level to get back in range. During this time you likely feel weak and shaky.

This chart shows all my blood glucose readings. In the first couple of weeks all my readings were out of range, but then they started stabilising, and now they are mainly in range.



When someone is newly diagnosed with diabetes in the UK, the full support network of the NHS swings into action. In addition to my GP, I have a diabetes consultant, two DSNs, and a dietitian. I saw all of these people in the first week, and I have ongoing support from the DSNs and the dietitian, who I can phone or email if I have a question, or need some help with my insulin dose adjustment. I have a recurring appointment with the consultant every six months. I’ve also been added to the system for annual screening for eye disease. The NHS also provides education programs for carb counting and insulin dose adjustment (one has the fabulous acronym DAFYDD - dose adjustment for your daily diet), as well as an excellent series of videos for newly diagnosed patients. All of this is free. And in Wales, where I live, there are no prescription charges for anyone (for people with diabetes in England, prescriptions are free too), so I don’t have to pay for the medical supplies that I now depend on every day. 

All of the medical staff that Eliane and I have encountered in the last two months have been unfailingly kind and supportive, even when working under pressure (like the first night in the EAU). It’s incredibly reassuring to have access to the resources the NHS provides. I would like to thank everyone there, along with my family, friends, and colleagues from work who have helped me get through the last two months.