Building Modern Software is Hard: I Help Teams Tackle It

Despite being called software engineering, the practice of building large scale pieces of complex software is still fraught with perils, both technical and social. Unmanaged dependencies arise out of seemingly nowhere. Team members experiment with a new technology that takes the down an exploration path that ends in disappointment. Cloud runtime platforms change their APIs resulting in applications that no longer scale as designed.

I’ve been researching and building production software systems for more than twenty years across a variety of platforms. I’ve worked on teams from tiny startups to some of the biggest tech companies.

It’s relatively easy to put together a small piece of software, such as software to generate and serve a weblog or a tool that uses publicly available APIs to give you the current value of your stock portfolio. Unfortunately, most software is far more complex - not only due to the technical requirements of the software, but also the way the software needs to interface with existing backend systems and the various individuals and organizations that will use the software.

In my role as a research staff member in the Software Governance group at the IBM TJ Watson Research Center in Yorktown Heights, New York, I develop technologies, methods, and processes that enable teams to build high quality reliable software. One of our primary tools to do this is better analytics of software development. We’re able to harness data from all aspects of the development process - requirements generation, planning, development, testing, deployment, and finally end user actions - they have all have potential areas to develop systems of insight to shed light on the development process. This allows teams to use real data to back up decisions made for their development processes and removes the guess work from software engineering.

Research Background

I have a rather interdisciplinary academic background that has served my research goals well. In 2009 I obtained a Ph.D. in Engineering and Public Policy and Computation, Organizations, and Society from Carnegie Mellon University. Engineering and Public Policy is an interdisciplinary program in the College of Engineering that focuses on the interconnect between the world’s engineering challenges, such as clean energy, using IT to support development, data privacy, and improving our nation’s electrical grid and how these challenges affect public policy decisions. It seeks to create engineers and scientists who can advise on both the technical and political realities of the issues. Computation, Organizations, and Society is an interdisciplinary program in the School of Computer Science at Carnegie Mellon that takes a broad view of computer science and looks at how the technologies we create interact with society as a whole, which can include public policy related issues, but also includes human-computer interaction, social equity, privacy, and social network magnification of technology.

This resulted in a thesis that examined the complex ways that Open Source projects work in the presence of commercial participation. We’ve moved far beyond the point where open source projects are developed by altruistic hackers in their spare time, if there ever was such a time, and my work examined how individuals, firms, and non-profit foundations worked together to produce value. It did this by examining the way that foundations, such as the Eclipse Foundation, work with member companies. Then delved into a technical analysis of how firms work with one another and how their presence affects individual developers. Finally, it looked at how individual developers work together through the lens of the socio-technical congruence metric.

This research used a variety of methods, including ethnographic observations, qualitative interviews, and a lot of data mining of archival software development resources, which led me to my current role in the software governance team at IBM Research. Currently I work on a variety of projects around IBM, the most prominent of which is JazzHub - IBM’s cloud based development environment.

Contact Information

The best way to contact me is via my personal email account, patrick@wagstrom.net. I’m also on Twitter where I’m @pridkett, but I’m not very active there anymore.