It is Just a Matter of Scale

It is just a Matter of Scale

Fractal geometry teaches us that the same patterns re-occur at large and small scale.



https://www.youtube.com/watch?v=G_GBwuYuOOs

In a previous post Standing on the Shoulders of Giants I talked about how increasingly we are developing standards that we can now use as building blocks to help us develop complex enterprise systems.  Most people know that for the past few years I have been an Architect for Boston College's EagleApps project where we are completing the core of the original Kuali Student product suite.

What you may not know is that for the past few years I have also been consulting on another project. That project just got a big boost from USA Funds.  

USA Funds Launches the DXtera Institute
See USA FUNDS News Release. 

Representatives for the DXtera are currently at Educause.

It should not be a surprise that these two projects that I have been working on have much in common. The standards that you use to bind together the different parts of a single system can and should be the same standards that you use to tie together disparate systems.  The semantics of the integrations are the same -- it is just a matter of scale.

Back to a Better Future

Back to a “Better” Future

How Boston College’s EagleApps is Flexibly Dealing with Rules

Throwing the Red Flag

On Oct 21, 20151 the Boston College EagleApps Enrollment team huddled to discuss adding a "flag" to courses that trigger automatic full time status for a student. BC had such a “flag” in their current home grown system and they were looking to add a “flag” like that to the Course Service. Several of us balked saying that adding a ‘flag’ is itself a RED FLAG to good service design. Like most schools the calculation of the full time status at BC is much more complicated than simply adding up credits. The threshold varies by program and whether or not it is a regular term or Summer. It also depends on whether or not the student was taking one of these special thesis/internship type courses. To understand why some of us took the stance we did we have to flash back nearly 10 years ago to the Summer of 2006. One year before the “Next Generation Student Services System” group joined the Kuali consortium.

1 OK the actual date of the meeting was Oct 22, 2015 but hey... close enough.

Was it possible to build a system that would meet all our needs?

It was July 11, 2006 and hot in MIT’s concrete bunker style Student Center. The workshop had been convened with a dozen schools in attendance. Small groups were chatting during a break from the intense modeling sessions. The sessions were centered on such mundane topics as curriculum development, student registration, financial aid and student billing. These are mundane to most people but to the group gathered here this was serious business. Some attendees came from the business side, Registrars and Financial Aid Officers. Some came from the technology side. All had decades of experience in higher education.

The workshop was the second in a series designed to explore the possibility of developing an Open Source Student Information System. Could a consortium of colleges join together and actually build such a system?  Was there enough commonality in all of our business processes to even think that building an open source student system was possible?

We’re not modest in our goals

My group had just finished modeling Financial Aid and we were listening to Chris Mackie, Associate Program Officer for the Mellon Foundation explaining why he was attending this workshop. The Mellon Foundation, he explained, didn’t really care about information technology, per se. The foundation’s core mission was to support the Arts, Music, Culture and Learning. However, we’re not modest in our goals, Chris announced, we want this group to change the way higher ed software is developed. From our perspective, he continued, billions of dollars is wasted on administrative systems that no longer support innovation and value.

Not “What” but “How”

We broke up and I went back to my modeling session on financial aid. I kept asking myself, was this Chris guy crazy?  What was so groundbreaking about what we were doing?  Sure we were trying to build an Open Source Student Information System but how was that going to change how higher ed software was developed?  Perhaps it might be cheaper and easier to modify because the source was open but it would still be just one more package to acquire, install and configure, wouldn’t it?  Most of the costs go to integration anyway so what is so “innovative” about that. Then I thought... Perhaps it wasn’t WHAT we were doing that was so important, perhaps it was HOW we were going to do it that was revolutionary.

We’re all the same except for all the “crazy” rules?

At the end of the workshop the groups came back together and I helped present our team’s results. Financial Aid in the US is on the surface very different than that of Canada. We struggled for the first day just to get past terminology differences. To our amazement, I continued, we found that all the universities actually shared the same basic processes, Canada and US. We called things by different names, did them in a different order, some skipped steps that others followed but the basic processes were all the same. We could agree on everything -- we could build a single system that would meet the needs of all the schools -- we’re all the same except for the “crazy” rules. To my further astonishment each group got up and reported essentially the same result. Everyone left the workshop convinced that the whole grand scheme was possible if we could find a way to handle each school’s own “crazy” rules.

Rules as First Class Citizens

Flash forward one year later and the Kuali Student Services System Program Charter is written and signed. The word “rule” appears 215 times in the charter. The configuration of business rules is called out as one of the major functions of the system -- clearly separated from the service contracts and the rest of the system:

Principle 5: Abstraction of Business Processes and Business Rules

Business rules and business process logic will be abstracted from the code base.

(p. 21)

The entire architecture of the system has been designed to move all that “crazy” rule logic out of the data objects and out of the main services and put them into their own services as first class citizens. This was the principle that some of us on the EagleApps team felt was being violated when we were asked to add a “flag” to the course.

Back to a “Better” Future

Boston College is now working hard to complete it’s EagleApps system based on the original Kuali Student source code, architecture and vision. After 15 minutes of discussion the EagleApps Enrollment team had come up with a “better” plan. Instead of adding a “flag” to the course object we would leverage two key technologies:

1. The use of several “rules” oriented services that allow us to cleanly separate the various aspects of the writing, configuring, viewing and executing credit load calculation rule.
2. The creation of a domain specific language (DSL) that will allow the detailed authoring of BC’s full time status rule by a technically minded business user.

This DSL technology is new and deserves some explanation. It leverages drools and was originally developed by Sigma Systems, Inc for the Kuali Student Accounts system. Boston College is currently using this technology to author it’s own Fee Assessment Rules and Payment Application Rules. Additionally Sigma Systems and Boston College just completed a POC leveraging the technology to calculate a GPA. Having done this, the jump to applying the DSL to load level is not that hard to imagine.

What the EagleApps team is completing is truly a Next Generation Student Services System. We are not following the traditional ERP patterns such as adding flags everywhere and burying the rules in code. If we did that we would end up with a “higgly piggly” mess of service contracts and code that would never be stable and around which no other schools could find agreement. The EagleApps team is not designing this kind of flexibility out of the goodness of their hearts. They are not doing this simply so other schools who want to leverage the source code can configure the system to meet their own needs. They are doing this because they know that the really cool thing is that this same ability will allow Boston College’s future self to easily refine their own rules and innovate educationally and help them produce a “Better Future” for their students.

Standing on the Shoulders of Giants

Standing on the Shoulders of Giants

At one level computer programming hasn’t progressed very much in the last 40 years. In 1974 in my first programming class at the Miami-Dade Community College I was told to write “Structured Cobol” to avoid GO TO errors.  Despite the invention of “object oriented” programming, “design patterns,” “inversion of control”  and other new paradigms for coding the fundamental issue still remains:

How can we write a big program to solve a complex problem that does not end up becoming a tangled mess of “spaghetti code?”

But wait a minute -- something has to have changed since 1974 (it is not just that the hardware is faster and I am slower).  It is just obvious that something about computer programming itself has somehow advanced significantly because we can do so much more NOW than we ever could do back then. 

So what is different?  

My answer has nothing to do with computer science but with culture.  In the intervening 40 years we have agreed upon an increasingly higher level of open standards. These are standards for accessing and executing certain kinds of operations.  Sometimes we call these “API’s” sometimes “protocols” sometimes “interfaces”  but the operative words are Open and Standards.  These have nothing to do with all those cool techniques that we as individual programmers value but they are the slow collective work of our culture.   Yes standards are often first inspired by or initially created by a small group of programmers but ultimately it is the culture that decides and sanctifies that standard by adopting them.  

Consider this progression

Each level builds on the previous and feeds the next.

(This rough diagram came from a presentation I did about Kuali and Pesc in 2011)

To borrow from Isaac Newton:

We can do so much more as programmers today because we are standing on the shoulders of giants.

When we write code we no longer write to a specific video card or hard drive manufacturer as I once had to. I remember arguments about the best printer protocols but today most of us don’t even know what protocol our printer is using.   We can no longer write to that low level but we don’t have to worry about it either.  This frees us to code at a higher level.  We can be more creative at that higher level solving bigger problems because the lower levels just work.  This only happens because we as a community have coalesced around open standards.

So... the big question is...  Are we really ready to tackle whole ERPs?

These are the kings, the actual business level applications that sit at the top of the stack.  These are the ultimate complex business problem that we as business programmers have been writing and maintaining spaghetti code for for decades.

In my view YES.  

We can tackle the application level because we have standard service contracts that were funded initially by the Mellon Foundation in two initiatives OKI and Kuali Student.  Over the past 10 years these seeds have allowed hundreds of people to put their hearts and souls into banging out, testing and refining a set of standard and open service contracts.   An often under recognized fact is that for Kuali Student the Service Contracts themselves were intended to be first class deliverables -- as important as the working ERP that we all labored to build.  See section 2.2 of the Kuali Student charter.

From the beginning these service contracts were designed to be adopted as standards for the community.  They now embody the wisdom and diversity of close to thirty institutions ranging from public community colleges to private research universities as well as international universities.   These contracts now allow us to design and build the ERP systems of the future.  This is true at least in the Higher Ed space. These contracts model all the things we care about -- semesters, curriculum, course offerings, registration and grading but we also model people and orgs and scheduling and logging and authorization and so on.  We can then use these contracts to build and release systems by and for Higher Education. In that domain at least I believe we are ready for that next step.  We can reach that highest level of interoperability.

More to come...  

In my next two blog posts I will discuss “how” Boston College is leveraging these standard service contracts to punch out pieces of code and then wire them together to form their EagleApps system.

I will also talk about progress on another project which hasn't gotten much attention... In this project we are using these exact same standard service contracts to wire together community colleges with disparate systems to allow them to communicate with each other and to career centers.  See these two press releases

• KPlus2 helps bring new colleges to the Kuali Community in a new way
• Mass.Community Colleges awarded $20M workforce grant (bottom paragraph)

As these two projects move forward we will indeed be Standing on the Shoulders of Giants.

New Things In Kuali Student/EagleApps

New Things In Kuali Student via EagleApps at Boston College

Last fall Kuali Student diverged (see previous post). KualiCo chose to start over from scratch to focus on cloud opportunities.  Boston College, an institution that was in the process of implementing Kuali Student,  out of necessity chose to continue on with the existing source code. In the role of architect on that effort I have helped put in place key architectural and process changes designed to move BC forward faster towards their goal. We are continuing with Kuali Student but we are doing so with some modified approaches.

First of all Kuali Student has been branded on campus as “EagleApps.” The primary goal is to create a system that will work for BC now and for a long time to come. A secondary goal is to provide a stable platform that other like minded institutions will be able use to implement their own solutions if appropriate.  As such the BC effort continues to identify likely configuration points and when done they will release the source code under an appropriate open source license.

Second, Kuali Student was built using a “Service Oriented Architecture.” Would “EagleApps” keep this orientation? In fact Boston College is perhaps more strongly committed to an SOA based solution. You see last summer, while Kuali was holding it’s community meetings that resulted in KualiCo, I was doing a “proof of concept” to test if we could address the significant integration challenges that BC (or really any university) faced by using those services. BC had already implemented CM and had started KSA and had a plan to implement Enrollment -- each was deployed using different versions of the source code. That POC showed that BC could use the Service Contracts to create “Bridges to Tomorrow” eliminating the need and risks to implement everything all at once in a “big bang” and to proof us against future technology changes.  Today BC is not just using those service contracts to bridge different versions of the software but to connect to their existing systems and their operational data store.

Today EagleApps is moving forward fast. We are leveraging the existing Kuali Student service contracts and source code to finish the original vision of Kuali Student as a modular configurable loosely coupled system.  We are also leveraging many of the procedures and processes pioneered by Kuali Student.  We are expanding on, adjusting and improving many of those ideas and approaches in a “new” and different manner.

1. The source code has been moved to BC’s private GIT repository with the intention of making it publicly available after our initial release.
2. GIT’s elegant branching features are employed to support the Kuali concept of parallel development with teams in South Africa, India and teams right here in the United States.
3. Like Kuali, the service contract documentation will be put on a public wiki to encourage comment, discourse and agreement on these core design documents.
4. Equivalent RESTful JSON bindings have been created for the service contracts supplementing, not replacing, the java binding and Soap/WSDL binding. Each binding has its own advantages and disadvantages.  Expanding the Kuali Student notion that we do not conflate the semantics of the contract with any particular protocol used to bind to the contract to any particular software language.
5. The service contracts themselves are maintained in a completely separate repository with it’s own governance structure. This was done to support other existing users of the Service Contracts including BC’s own web-service team and to prepare a path to sustainability.
6. The implementation source code layout and structure has been reworked to speed development, i.e. developers no longer have to rebuild and deploy the entire application just to test a tiny change they made to a single small feature.
7. KRAD is now deprecated in favor of move Kuali Student made towards AngularJS for UI development.
8. As was Kuali Student’s intention, each service is built and deployed independently in it’s own war file, i.e. no wiring together of services underneath as may have happened during prior development.
9. Similarly the user Interfaces and other applications also are built and deployed independent of their services. Stated another way, all applications must be able to access the services remotely across the wire.
10. Integration with RICE is deprecated with alternatives found to decrease our dependence on a middleware product given changes to RICE.
11. The loading tools developed by Kuali student have been expanded so we can load most data directly via the services instead of relying on SQL inserts.
12. A particular school’s configuration data, i.e. types, and states and other school specific data, is managed in easy to use spreadsheets that BAs can update and then directly load to configure the system themselves.
13. Use of full coverage AFT’s, Automated Functional Testing, is deprecated in favor of rigorous programmatic testing all services coupled with an AFT “lite” approach for the UI.
14. All services are instrumented to facilitate performance and load testing, production monitoring and troubleshooting.
15. The code generation techniques that were first developed by Kuali Student have been aggressively pursued improving development velocity. A service can now be implemented in hours instead of weeks.
16. Business, validation and authorization logic has been moved out of the application and placed behind the service. This is done to ensure data loaded via the services will work when called up in the UI and to make it easier to write and test new screens.
17. All code changes are accompanied by a corresponding unit test to prove the logic does what it is supposed to do.
18. All code commits are accompanied by a collegial code review inviting all peers to inspect, comment and learn from each other.
19. Over dependence on Spring Beans in general and on RICE’s configuration scheme has been deprecated in favor of a more pragmatic approach based on the kind of configuration being done.
20. Each feature has been split into a service component and a UI component so we can run dual agile teams developing the services first then the UI.

In future blog posts I will be digging deeper into some of these technical and process changes. In the mean time Boston College’s EagleApps will be “screaming” forward.

Two Paths Diverged In the Woods

People who know me have probably been wondering why I have been so silent about the whole KualiCo issue.  I’m typically very opinionated and am not afraid to speak my mind.  The reason is that I have been thinking very deeply about what is happening.  I have come to the conclusion that Kuali Student and KualiCo were created with two very different goals in mind.

Kuali Student’s main goal, from its charter, is:

To support a wide range of academic and related business processes, including those that cross departments, systems and institutions, in ways that work best for each institution, and make it easier, faster and less expensive to change existing processes and introduce new ones.

KualiCo’s goal, from its mission statement, is:

KualiCo’s mission is to provide amazing software that lowers the cost of higher ed administration and improves the learning experience for students, faculty, and staff.

Now those are very divergent goals.  KualiCo with its very laser focus will hopefully achieve success and volume.  I hope they do because KPlus2 plans on selling add-on products through them.  However, as dictated by their mission and as reinforced in their marketing they will have to create a system that is simplified and streamlined (their words not mine, see KualiCo’s website).  Their student system, when finally delivered, will most likely meet most of the needs of most of the schools in the United States.

But I also believe that many institutions are not like most schools.  These are the Institutions whose faculty and culture require that their systems be much more flexible.  These are institutions who see that their ability to compete and differentiate themselves is not based  solely on saving dollars. These schools differentiate themselves by their ability to experiment educationally and they rely on processes and systems that enable this flexibility.

From my perspective there is room and a need for both Kuali Student and KualiCo.

What's your perspective?