This article appears in K. Aberer, D. DeRoure, O. Dousse, D. Fain, G. Flake, F. Heylighen, D. Pennock, W. Shen, P. Thiran. Neurons, Viscose Fluids, Freshwater Polyp Hydra - and other Self-organizing Information Systems. IEEE Intelligent Systems, Trends & Controversies, 18(4), Jul/Aug 2003.
The term grid computing has previously suggested a world of networked supercomputers, Beowulf clusters, fat pipes, and petabyte storage. But now the 'grid problem' is defined as 'resource sharing and coordinated problem solving in dynamic, multi-institutional virtual organizations' [1]. High-performance computing might once have focused on accelerating scientific computation, but contemporary grid computing is also about accelerating the scientific process. It is the infrastructure of e-Science, and indeed e-engineering, and potentially e-many-other-things too.
The notion of .the Grid. is an analogy to the electricity power grid - you can plug into the common interface to tap its computational power. The middleware that implements it, de facto the Globus Toolkit, hides the heterogeneity of the diverse computational resources. In principle, the middleware makes it easier to cross not only operating system and version boundaries but also organizational boundaries. This is significant because the computational power can come from different power generators.
Grid infrastructure and applications are typically service oriented, and in the last year or so, the middleware effort has fallen in line with the prevailing movement to Web services to enjoy a degree of industry-standard interoperability. This takes the form of the Open Grid Services Architecture [2], an enhanced Web Services model created to meet the grid community's specialist requirements.
A huge number of grid projects exist, fueled by national funding programs such as the UK's US$500 million e-Science program, which reaches across a spectrum of research disciplines. Every time we embark on a new project, we would like to reuse and repurpose the data, services, software components, resources, and indeed knowledge from our own scientific communities and others' previous projects. So, where we once needed the Grid to hide the heterogeneity of computational resources, the new Grid problem is hiding the heterogeneity of the bits and pieces needed to quickly and easily assemble new projects, or even new grids. The vision is a generically usable e-science infrastructure, comprising easily deployed components whose utility transcends their immediate application. Our goal is the automated construction of the desired services, adapted to the current requirements and circumstances. This is the self-organization we seek - the assembly and adaptation of Grid services.
Underlying any form of automated composition we need an infrastructure where all resources, including services and workflows, are adequately described in. machine processable form; that is, knowledge is explicit. Semantic Web technologies provide the infrastructure. This has led us to the Semantic Grid [3] where we apply those technologies in grid-computing developments, from grid infrastructure machinery (such as the Globus Toolkit's grid services) to grid applications. 'Semantics' permeates the full vertical extent of the Grid and is not just a semantic layer on top; it is semantics in, on, and for the Grid. Some of these Semantic Web technologies are ready for immediate deployment (the Resource Description Framework tools, for example), while others are on the research agenda. Realizing the Semantic Grid vision involves bridging two rather disjoint communities, each with its own standards process. At one end is the World Wide Web Consortium and the Semantic Web research community; at the other is the Global Grid Forum and Grid research community. GGF now has. Semantic Grid Research Group, chartered to help the Grid developers apply established metadata technologies while tracking, for example, the evolution of OWL (the Web Ontology Language) and associated tools.
The Semantic Grid is a complex and large-scale piece of machinery. Some aspects of the system are beyond centralized control. Looking inside will let us see the opportunities for self-organization.
The Semantic Grid's computational fabric comprises interconnected processors - perhaps with some dedicated networking locally - that are globally interconnected through the Internet. The Internet is undeniably a large-scale decentralized system. In fact, the Internet is an adaptive system, comprising simple components with local knowledge that together provide a global network service that copes with network component failure. Above this, we might consider the Web's machinery to be a large scale decentralized system, but I dispute the scale of distributed processing - a typical Web transaction involves just a server, a browser, and a proxy or two. The evolution of the deployment of the Web infrastructure, however, is an example of a self-organizing system. The large-scale decentralized system in the Web is actually its content, the global linking infrastructure that has generated much analysis in recent years. The Semantic Web gives us a much richer way to describe the associations within Web content, transcending the limited expressiveness the simple navigational linking model provides. As the decentralized metadata grows, it is interlinked by the resources it describes. This is a critical basis for self-organization of the Semantic Web content.
The Semantic Web assumes a service-oriented model, such as Web services or Grid services - or agents, which are producers, consumers, and brokers of services. We are beginning to see the application of Semantic Web technologies within that infrastructure, in multiagent systems, and through Semantic Web services. This emerging, semantically rich service-oriented infrastructure is an important large scale, decentralized system in our Semantic Grid. It's the last to emerge, but it fits in the middle. it is the semantic middleware.
These three ingredients - communication or computation, semantic middleware, and semantic content - provide opportunities for self-organization that support the vision of the self-organizing Semantic Grid.
The first is at the level of job submission and control over the computational fabric of large clusters or supercomputers, and the aggregations of these into grids. A single parallel computation can be predictable, with a known shape and extent, or dynamic and evolving. When you run an entire user community on that fabric, with myriad, diverse computations accessing distributed data, you have what is essentially a huge optimization problem. With job scheduling currently managed by individual schedulers, we should enjoy the benefits of self-organization benefits as we scale up. We would surely benefit from the fault tolerance that self-organization could provide because component failure is inevitable in systems of this scale.
The second opportunity is at the service level because this level has some homogeneity and scale and is where the many grids are beginning to meet to form the Grid. The organization task here is discovering and binding the services together to meet the requirements. This might be highly engineered, with prescribed workflows, for example. It might also be highly dynamic, with opportunistic use of services that have been instantiated and local data. The picture bears some relationship with peer-to-peer. Again, we have. massive distributed optimization problem.
The third opportunity relates to the information and knowledge that is the 'stuff' of e-science - the inputs and outcomes of the computations and experiments, and the descriptions of the processes that created them. Here we have scale, and we need to join the information with the processes that consume and generate it. Science is a collaborative process, and we might also view matching information to individuals as a process of organization.
But is the Grid really a large-scale distributed system? Anticipation of distributed processing on a massive scale has motivated its development. But currently many disjoint grids exist, just as many networks once existed that eventually combined into the internetwork known as the Internet. We can foresee the current grid islands similarly aggregating into an 'intergrid,' known as the Grid. Surely this kind of aggregation was the reason for the Web's clichéd 'exponential growth'. Once the Grid aggregates, we will have a large-scale distributed system with highly engineered, complex infrastructure machinery. Internet or Web nodes forward packets and documents; the Grid's nodes will do that and some computation. We do not yet know how much intergrid coupling will occur at the lower level (Web couples using the HTTP protocol) versus at a higher level (as with the integration of scientific services).
Of course, the notion of systems that can self-diagnose and self-repair is compelling, and the Grid community has recently shown some enthusiasm for autonomic computing [4], drawing inspiration from the autonomic nervous system. We are already achieving some degree of fault tolerance by engineering an infrastructure that monitors and repairs faults. This touches a key point: Do we need a self-organizing system, or can we engineer the necessary behavior? To put it another way, who needs slime mold when shell scripts will do? While our computing infrastructure is organized as an aggregation of managed resources, we can achieve a degree of centralization in control and coordination of the function of those resources. We are not obliged to be decentralized. Self-organization is attractive when it offers some optimization without risk - as Web caching does. but can we deploy it in a world where code is handcrafted for performance?
Of course, this vision assumes we have a Semantic Grid and shares some of the usual Semantic Web obstacles. It requires a metadata-enabled world, but where will the metadata come from? Except where classifications and domain-specific ontologies already exist, the Grid community must agree on the schema and ontologies. What will motivate this effort? I would claim that with the Semantic Grid, we have a fighting chance. The Grid community is a coherent, organized, enthusiastic community with real application drivers and could genuinely benefit from semantic interoperability. It will also stress Semantic Web technologies because it demands an extraordinary degree of automated interoperability, such as ontology mapping, and large-scale performance.
The vision also assumes masses of computational power, but we can be somewhat confident in this trend. Apart from more powerful processors, we will simply have more processors, emphasizing the issues of scale. Consider the provocative amorphous computing example of embedding processors in materials (for example, 'concrete by the megaflop'), which, incidentally, insists on self-organization (see www.swiss.ai.mit.edu/projects/amorphous ). Notice also the broad similarity between the service-oriented adaptive Semantic Grid vision and the service-oriented adaptive ubiquitous-computing vision. Several services must come together locally to meet our needs on a distributed architecture where devices come and go.
Let's push that autonomous processing a stage further and combine self-organizing services with self-organizing information and knowledge. Our self-organizing Semantic Grid is now a constantly evolving organism, with ongoing, autonomous processing rather than on-demand processing. This evolving, organic Grid can generate new processes and new knowledge. Consider, for example, a genetic approach to creating new workflows to solve a scientific problem. Imagine watching this autonomous, self-organizing Semantic Grid in action, visualizing the information flows in the system. It might be very similar to watching a bunch of e-scientists at work. But surely individual scientists bring problem-solving insight to the process? Maybe, but on the macro scale is there much difference? Remember, this is Grid computing, and we have the facility for large-scale processing. Additionally, people are slow. The DNA microarray has caused a revolution by permitting masses of parallel experimentation, as has combinatorial chemistry. Experimentation in silico is another such shift. Some Grid applications can and will absorb as much computational power as is available to produce better or faster results, but we will also be able to do more parallel experiments more quickly, monitoring and steering them dynamically.
I've shown that we have some opportunities for self-organization, but the current infrastructure might not be ready for it. And I've shown a vision of a self-organizing Semantic Grid that serves the scientist by assembling services on the fly and could even embark autonomously on scientific discovery. The challenge is to determine which parts of our self-organization are highly engineered, perhaps through the proactive behavior of agents, and when to let go.
1. I. Foster, C. Kesselman, and S. Tuecke, 'The Anatomy of the Grid: Enabling Scalable Virtual Organizations'. Int.l J. Supercomputer Applications, vol. 15, no. 3, 2001.
2. I. Foster et al., The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration, Open Grid Service Infrastructure WG, Global Grid Forum, 22 June 2002; www.globus.org.
3. D. De Roure, N. Jennings, and N. Shadbolt, 'The Semantic Grid: A Future e-Science Infrastructure,' Grid Computing: Making The Global Infrastructure. Reality, F. Berman, A.J.G. Hey, and G. Fox, eds., John Wiley & Sons, 2003, pp. 437-470; www.semanticgrid.org.
4. IBM, An Architectural Blueprint for Autonomic Computing, Apr. 2003, www.ibm.com/autonomic.