The phrase "the more things change, the more they stay the same" has been on my mind lately. Computers have become such a part of our lives that we can’t imagine life without them. Just a few years ago it wasn’t uncommon to hear that Blackberrys called "Crack-berrys," referring to the addictive nature of having email readily accessible. Today we even have a thing called "Computer Addiction" that people can get treatment for.  The children coming of age in this era they are the most informational connected generation we have seen.  Considering the architecture changes, the changing expectations, and rate of adoption the future definitely has a more interactive and mobile look to it from a computing perspective.

I recently read where mobile devices have outsold traditional computers for the first time in the 4^th quarter of 2010. Apple has been reporting sales growth while companies like Microsoft, Intel, and other companies are reporting lower than expected returns for the same period. With certain product releases coming in 2011 the anticipation is the sales trend will only continue to grow the gap.  As we see the sales trends change and more companies trying to capitalize we look to anticipate its direction and build products viable for today’s market and tomorrows.

To anticipate the direction we can start by focusing on a brief history of the mobile devices. Consider that Microsoft was an early player in this market. Compaq iPaq’s, HP Jornada’s, and others were touted as Pocket PC’s. Toshiba had one of the first tablet PC’s I remember. It even had a built in camera but the unit was very heavy. Microsoft envisioned "smart devices" and for a while had produced marketing as such. I remember they envisioned the device could be replaced and your configuration auto-magically restored. They had great vision and they dominated the early market. But while they were an early endorser and participant in the mobile field a couple of miss-steps and lack of innovation later they were behind.

It’s arguable that widespread adoption started to change with the acceptance of the Blackberry. Users were getting email connecting in ways they really hadn’t before. It wasn’t long before next up were the expectation to be able to review attachments to email. Having the internet on a mobile device wasn’t far behind that and the expectations began to speed up.  Why? because the adoption rate improved. Users saw immediate value in the functionality of these devices. But devices had different purposes. Blackberry’s did email while pocket pc’s handled calendars and other basic functionality.  I remember at one point having so many devices I felt like a techno-nerd version of Batman. While this was going on Apple envisioned the iPhone. Apple developed the iPhone in quiet and when they released it changed the market.  The change was significant enough that the carrier they worked with to support the device was overwhelmed for a time with new customers. It seemed like overnight they met and exceeded user expectations, and made a giant leap forward. Others began to follow the trend.  

User interface expectations are certainly being affected by changing expectations. How long did companies toy with keyboards until the iPhone changed the game with the popularity of its touch interface? A touch interface for a mobile phone had not been accepted until then.  Apple tried to compensate for users comfort by adding "clicking" sounds to the iPhone. But the hardware wasn’t the only innovative aspect. They innovated software are delivery as well.

The layout of the Apps wasn’t entirely new. Icon short cuts on a desktop have been around in the Mac and Windows worlds for years but the operation or implementation around the apps was. Users were able to use the devices to quickly check what they deemed the most important things.   Another expectation is the speed that these devices are expected to operate at. Long load times are not acceptable.  In addition to load times connectivity has become a key factor as well, a key contributor to the onslaught of the battling ‘G’ advertisements and related devices.  

Delivery handled through iTunes and working directly with the Apple company remains the only way to deploy applications. With the combination of hardware, software and deployment the entire platform was innovative and users liked it.

With a great rate of adoption and renewed interest in the market other players have been working to be more competitive in this market. For example, the Droid and Microsoft’s Metro concepts are two emerging or re-emerging market competitors.  With all the various players history in some ways begins to repeat itself. As they have gained more market share and their sales increased as well technical complexities re-emerge.

We still have a familiar challenge though, remember the old Mac vs PC days? Well we are there with mobile. We see different operating systems, different carriers, lack of interoperability and different devices. Consider that Adobe’s flash won’t run on iPhones. These types of complexity have a strong feeling of déjà vu for some of us. Only now we have added the extra complexity of Different networks carry different devices and different operating systems.

The innovations in both the hardware and software will continue in the space Apple has defined for a while. We are also seeing a repeat of some of the same hardware and operating system issues that have plagued IT for years.  What is different is that the adoption rate is continuing to grow. Watching over the last several months I see more executives and other carrying tablet PCs to meetings instead of the traditional notepad.  

Innovative development on the mobile platform will remain costly in some respects. Developing for multiple operating systems and different devices presents many challenges. What’s different today is that there is more of a drive than in years past to build these solutions. There are and will be tools that enable development for devices as well as across multiple platforms. However, those tools will have limitations and it will be a challenge to truly innovate through them. While working in the native system means developing different code for the same app to work on the different systems. Architecting a solution in mobile has to take into account the various considerations. Companies have to decide if they want the expense of creating an innovative app for the mobile platform or just have an app for the platform. This has a significant cost difference especially if the app has to be deployed to multiple devices.  

As customers decide their goals and directions in the mobile space it will be important for architects to use the tools available to them. The use of design patterns and object oriented techniques will be of paramount importance going forward for the software side of solution.  Creating a scalable solution for the growing functionality needs of mobile users will be critical. Considering that the hardware of the PC has evolved at a much slower pace scalability will have additional challenges in the mobile platform.

Creating a scalable solution is more challenging with the frequent release of devices and the secrecy surrounding them. Many of the tools on a mobile device have been tools available on a traditional PC. Going forward the hardware is starting to move into truly new areas.  For example, talk of the iPhone 5 and the capability of it having Near Field Communication capabilities have been going on for months. Talk has already started about functionality of the iPhone 6. Architects will be able to help customers prepare for not only the next deployment but the one after that.

The mobile environment is a market that companies cannot continue to avoid as it has passed the tipping point of adoption. But those same companies have to realize where mobile is in it maturity. Companies will not be able to build an app, deploy it, and then forget it. These apps are living in an every changing world and will need maintenance to continue operating effectively.  The architectures supporting the apps and contained within the apps must be able to scale to meet these needs.

The mobile environment is changing frequently and stepping forward in leaps we haven’t seen in a while. Developing solutions for customers means considering all the factors and leading them to understand the environment. Bobby Knight is probably as polarizing a figure in college basketball as there is. For all the negative about him later in his career, he is regarded as a great teacher of the game. It’s one of those lessons that really apply here. He said, I am paraphrasing, "we have to focus, by focusing it allows us to notice trends, recognizing trends allows us to anticipate, and that leads to action."

Designing systems, interfaces, back end engines, databases and other system related items takes a blend of creative and technical acumen. IT professionals love to solve problems with their blended skill sets. When an algorithm is developed we sometimes even share those ideas and algorithms so it can help other systems. One case in point is my colleague Jon Taylor’s recent blog about the Observer pattern and .net event models. (Check it out of you haven’t yet-great information.) Design patterns in themselves are previous solutions that give us a solid base to develop a new algorithm from. But how many times and in how many ways have we overdesigned something?

I remember a bad decision I made at the beginning of my career in overdesign. A customer wanted a backward compatible query building wizard for their staff to use that had little technical knowledge or database understanding. However, the trick was the ‘management’ staff consistently wanted to ‘adjust’ database table and field names. The data in them wouldn’t change but the name needed to change. They were adamant about this. In coming up with a design we looked at using SQL Server ID numbers to store. This way we could recreate the same query with new names and we wouldn’t have backward compatibility issues. When built it worked great everyone had what they needed. Until a new version of SQL Server came out that instead of updating table names dropped them and re-created them. Slap the forehead time. I didn’t consider the impact or fragility change would have on the system. I focused on one direction to much.

Instead of looking to a simpler solution I went to a more complex one, or better said as an overdesigned one. I have never forgotten that lesson and I find all kinds of ways to apply it. Over the years this has turned into a philosophy and phrase ‘evolution not revolution’. Revolution is not limited to overdesigning the back end system it could be overdesigning any part of the system. In this context the concept of revolution for change is a bad one. Some of the aspects of revolution I am referring to are the destructive nature, the amount of change that has be absorbed, what can’t be changed back and the time to adjust to the changes. When your users are feeling the impact of a system revolution is this a good thing? What causes them to feel that?

 We aren’t talking about users feelings at the deployment of a system. We are talking about the users feelings over time. One example, I heard company say they couldn’t adjust the size of a banner frame for their customer because they had built it custom. It would take to many hours to change. That small issue was the start for the customer losing faith in the product and the team that created it. Eventually that team lost the business. Why? They created a revolutionary new banner frame that was great. But when the customer needed change they couldn’t accommodate efficiently and cost effectively.

When architecting we have to be able to break down a system to its critical components and work forward. We have to keep in mind that the parts of our systems need to evolve and grow. To best serve our customers we have to be diligent in evaluating if our design is evolutionary or revolutionary. Thinking of the quote ‘an ounce of prevention is worth a pound of cure’ what are some simple ways to prevent overdesign. Collaborate; working alone in a vacuum is not always a great way to a solution. One of the reasons collaboration is such a great tool for teams is someone is always looking for a simpler way. If you aren’t working with a team or before you review with the team, ask yourself am I designing to the requirements or am I trying to add value. Sometimes in trying to add value we get so excited about what the system could do we forget that the customer may have never wanted that.

Prevention begins early in the process. Understanding the workflows and business processes associated with a system is critical. Architects, testers, developers, and customers all need to be on the same page. In gathering requirements we have to dig to the deeper process to design a quality system. With an understanding of the business processes, the entire development process begins from requirements on down reflect the beginning is a base.

Realize that for many systems that have heavy interaction, users need time with the system. This is important because I am frequently surprised with the direction a customer will drive a solution over time. It is so important to consider this during design as well as critical to take the time to design. It surprises me that even today how many developers and team almost go straight to code.  Take the time to focus on the base and look for evolution. For architects and developers design patterns are great if it fits and gives your customer what they need. Consider that the only constant is change. Do you really need that whiz bang new pattern or will something simpler work better?  

Create a system that can evolve with the customers’ needs over time not revolutionize today forgetting about tomorrow.

A common refrain in IT is to "begin with the end in mind." It is one of those refrains that has been repeated so many times that it quickly becomes ignored. For systems that collect data, "the end" may be very complicated. Some data collected can be presented in reports and that is sufficient. Reporting is such an easily understood term and used so frequently it often hides complexity. It’s in that complexity that frustration for system engineers and customers alike begins.  Because of that, complexity reporting is often not a cost effective means for exploring data. Complex reports can take a significant amount of time, (translate that to cost), in order to complete.  When users are looking at the data to see what trends or interesting anomalies appear, reports just aren’t efficient. We begin by recognizing that "the end" can be as complex a system as the front end collection system.

Architecting an analysis system can comprise many parts ans some of the most overlooked and underutilized are OLAP tools (Online Analytical Processing). As such, optimizing those tools and working with users to capture the areas the tool needs to work with are important.  It’s that OLAP part of an analysis system that we begin to explore in a bit more detail.

So to begin working with OLAP, two terms will immediately need to be understood--dimension and measure. Working off of a graph diagram is the best explanation to start with. Looking at Figure 1, notice the X–axis of the graph. In OLAP, that axis will be called a "dimension". The calculation that occurs at each notch in the access is called a "measure".  That in mind, figure 1 shows location as the dimension and debt amount as the measure.

Figure 1.

Using the same basic graph model, imagine that we add a new dimension debt type. So now the intersection that points between the dimensions debt type and location show our debt amount measure. The great thing about OLAP tools is that depending on the particular tool, they can look at many measures at the same time, can flatten a dimension out, and look at all of one and a particular point or notch on another dimension. So, why are they called cubes?

 Figure 2 shows the addition of the Calendar dimension, essentially time limited to the granularity level of day. With that dimension our graphical representation now resembles a cube, hence the name. While graphically we can only represent 3 dimensions clearly and easily, an OLAP system can represent hundreds of dimensions.

Figure 2.

OLAP tools provide different types of interfaces that allow users to simply drag, drop and click to explore data. Some tools like SAS Enterprise Guide can even graph dynamically while the user is exploring. Robust tools like this provide a good user experience and enhances the analysis system. Many systems like SQL Analysis Services connects directly to SQL Reporting services, allowing for reports to be built quickly and easily in addition to the normal canned reports. So, what does an architect need to consider with a basic understanding of what OLAP does for users?

An OLAP system basically represents hundreds or thousands, depending on the number of dimensions and measures, of queries working against the data simultaneously.  In processing, systems like SQL Analysis server offer different processing options to do as many pre-calculations as possible. These processing options are HOLAP, MOLAP, and ROLAP, but let’s not get into the weeds. What we need to know is that the system is going to perform some level of pre-processing and store that pre-processing. To do that pre-processing, and for that matter, processing, there are data structure designs that work more efficiently with OLAP. Some will refer to these models as Star and Snowflake patterns. What architects need to look at is that the data model for the collection may be differently optimized. Don’t get into discussions of normalization here, as it doesn’t apply. Both the data models can be normalized but still not be optimized for their working intent. Normalization does equal optimization for all cases.

Looking at figure 3, the data model shows a table labeled with the prefix "fact." The fact table in an OLAP model is where the values to calculate measures are usually stored. Keep in mind that a measure is an individual calculation and the facts are the values that enable those calculations. To help the OLAP system work, a group of foreign keys relate out to the tables with the prefix "dim." The dim tables may be single tables as represented with the debt type. Or they may be hierarchal as represented with the Calendar dimension.

Figure 3.

What creating a database model like this allows for is the OLAP engine to process quicker. Creating this model may result in ETL routines being created, which is a benefit, as dimension data can be transformed from being collected one way to another in the OLAP data model. This provides even more flexibility towards what we can represent, as well as interjecting new considerations. How often do we move the data? Do we copy the data or do we move it? An analysis system has to consider all of these options.

OLAP can also be a data source for the front end interface and provide new dashboards and functionality to the collection interface as well. Visual Studio.net for example, has quite a few capabilities in this area that will be in another blog coming soon. Getting back, this allows analysis systems the capability of being an integrated system in new ways with the collection system.

If you see the value of OLAP, look at it as another tool for the analysis system. Yes, it can be a replacement for some reports. However, a solution will still include canned reports, as we are always going to need those.  A full analysis system is going to have more components to it and be fully thought out. With this end in mind, if you were to design the exploration, reporting, queries, and analysis you want a system to provide, how different would the collection system be?

Many systems are designed from the point of view of the collection system. After all, that’s what most users will see and work with. So we focus there. Only later to realize we want to analyze data and we have the wrong level of granularity or we are missing a variable. Many know that the cost and time increase significantly when changing a constructed system versus at design time. Have you considered what the cost is of building the systems in the wrong order? Is the tail wagging the dog?

Many application development companies regadless of web development or windows development want or need to implement search functionality. However, it is a commonly underestimated function and it continues to evolve over time. Interestingly users want search to have minimal to no interaction while having a maximum result. With data and collection systems becoming more and more complex this becomes and increasingly difficult challenge. I remember a system I was architecting for a customer where the customer wanted to enter a DNA result that consisted of an 800 to 1600 character string into a web application and have it search a database using an algorithm providing scored search results. The customer was convinced that a basic desktop machine would act as a server and be able to conduct the search against a large database efficiently. The production architecture needed to support the customers’ performance requirements was a High Performance Computing hardware environment.  Like many customers they didn’t understand the complexity of certain functions. Thinking through this topic recently had me researching how functions in systems and their architectures evolved.

Architecting a system today has many facets, and search certainly is a prominent one. Searching for information is not a new concept but a heavily evolving one. Once computers evolved beyond just basic mathematics and started capturing, storing and manipulating other data the need for search began. Early systems collected data that was somewhat structured in files and databases. Search functions found data quickly within those structures. With the development of relational databases and more complex data capture search the tools for search had to grow. Also the acceptance and use of computers was growing and more and more.
Architecting search within a system has consistently had to recognize simultaneous evolutions. Database tools added the ability to index tables to help search perform better. Search appliances like Wizards emerged for more technically savvy users to pull data from a data source. Multiple levels of searching complexity were emerging. While these searches largely dealt with structured data stored in systems, at the same time this evolution was occurring what cannot be ignored is the emergence of the internet and its impact on search. 

Early on companies like Yahoo profited on the simple concept of locating content. While this wasn’t structure data as in databases internet standards of things like meta-tag’s and other items made it possible for users to find content early on. Searching on the internet allowed users to enter terms and content related to those terms would be returned. Later companies like Google would improve the algorithms and set that industries standard for a time. E-commerce companies were also integrating user shopper experiences with search as a means of driving revenue. So while a user shopped for shoes, related items and previous shopping items would appear in the links and advertising throughout the system. While the motives were different the capture of information and providing relevant data back is essentially an implied search. The evolution of the internet and its potential was impacting local systems.

Users’ expectations were changing as the interaction was to enter in a few terms and that brought back content they wanted to see. At the same time computers continued advancing in hardware and use. Pictures, Videos, art, music files evolved to become more common to be stored on systems. In fact digital has become so big that companies like Kodak have stopped producing film based cameras. Users have embraced and ran with the lower cost and portability of digital media. This new media has presented a new challenge and forced search to evolve in multiple ways again.

Architects and systems were faced with growing use for search.  Users were searching as an exploratory exercise as more complex data and more types of data were being captured. Allowing for the advancement of tools like Online Analytical Processing (OLAP) and reporting tools. Users weren’t looking for specific data as much as looking to see what trends might appear in the data. These tools while technically complex have easy to use interfaces that allow users to review and analyze data. The complexity lies in the architecture and backend. The emergence and development of these tools was a move from appliance parts of a system to search to a full blown system of its own.

Users now expect applications to be able to search both structured and unstructured data. They want to give as little information as possible and quickly find very relevant search results. Algorithms and techniques for searching continue to advance because they must--including incorporating e-commerce like changes in the system and having subtle changes help the customer get to the results they want more quickly. One of the many reasons unstructured data evolved was not only digital media but mobile devices.

This latest evolution has occurred simultaneously with the acceptance of mobile devices. Now users have a high level of portability and connectivity to data. These mobile tools work quickly using touch screen technology and other key changes that impact the user experience for working with data. This has resulted in a need for better performance and system architectures that incorporate different devices, connectivity, and desired results.

Today’s cutting edge searches involve grabbing information from a part of a picture and searching for related information. Searches that work from audio files or live audio and provide related information quickly on portable devices is another technology that has been developed. Users want more with less required of them, resulting in more complex algorithms and models for searching.

Successfully architecting a system means taking a lot of factors into consideration. A successful solution can't overlook what the implementation's search functionality has within an enterprise system. Architecting search as a part of a system today means taking many factors into account. Understanding the user’s expectations and desired results has become critical to the successful use of a system. What devices are targeted for use, what is the complexity of the data, what type of data, and other questions like these are all key to get answered to develop a successful search system. Working with customers to identify the business rules that lead to implicit and explicit searches is important as systems more and more are expected to show relevant data.

The Information Technology field has to have one of the highest rates of evolution of any field. A friendly warning for College Students, if you don’t like learning and discovering choose another field. Over the last 10 years the evolution of the web has been constant. Today we have information flowing to multiple channels, more complex information being captured, and more data being provided to users. With all of the content and information available it is no surprise that finding that content has had to get more complex as well. Optimizing your web site or web application for search engines is getting more and more complex. One aspect to look at is a subtle one. Ensuring that your site map and your keywords are captured, architected, and developed to work together.

A good BA is worth their weight in gold and early on in the requirements and discovery process capturing the keywords can really help the development of your tool. Keywords are a known importance to optimizing your site for organic discovery by Google, Bing and other search engines. There are tools dedicated to keyword mapping to show how your site will be captured by a search engine. What the keywords can’t be, however, are an afterthought to the development process. Keywords are concise definitions of your web site. Like the advertisement on television for a popular clothing retailer right now, the tag line is “Modern. Southern. Style.”. In three short concise words they define themselves. Even the government has taken to this “Safer. Healthier. People”. Keywords have been around for a while and we all know about them but I bring this up to discuss how we focus on them and use them.

A BA can use keywords to focus requirement sessions, the architect on the site map and architecture for the system, designers to ensure the colors layout user experience match the keywords, developers for for the folder structure, and testers to make sure they got it right. Now some would say that keywords should be derived from the requirements and the experience the company wants for its customers. Which is a great point that opens a question, are the creative people that can help write that copy and help getting involved early enough? Once the keywords have been defined so much can be based on them. The point of this column is architecture so lets jump there.

When the site map is being determined and the layout of the site designed/architected keeping the keywords in mind can really help. It is a common best practice to have a site-map on your web site. Many web sites have several versions to ensure they are read by the search engines. Ever added an XML web site document to ensure Google would read it? So using your keywords in various other locations can greatly assist your website.

If your keywords define your site and its content then shouldn’t your page titles include these keywords? With our keywords in the title another step is to ensure that we use the keywords in the URL. For example, instead of www.sitex.com/en/ we could include keywords www.sitex.com/keyword-keyword/. Not only is this more descriptive for the user the search engines will jump up the importance score. Why does this need to be part of the site map? If you are going to include keywords as part of the URL and folder structure the developers need this info to focus on. So that means knowing the site map before the pages are developed so they can use this information to their greatest value.

Considering the search engine will use the links on the site map to crawl the site, using keywords would help raise the score wouldn’t it? Getting into Canonical URL is a little beyond the scope of what we are discussing here but is a topic you might want to look up as well. While it may seem simplistic at this point in time of the internet’s evolution, keywords are still and will remain and important part of content discovery. Understanding how to re-engage on the importance of keywords and their use can help prepare for future evolution of the web.

Ever hear of the concept of ‘the semantic web’? Today a user views pages for information gathering and capture for activities like travel. With the sematic web, pages will interact in a more automated fashion reducing the amount of work a user does. As the web continues to evolve the potential for keywords to grow in importance is still relevant even considering their long history. The tie in to the site map becomes more important as desired functionality evolves. The key to scalability will be planning today for what is coming tomorrow. Preparing for tomorrow begins with looking at the process, collaborating, and working to the future. Don’t pass over the simple things, they just might be the key to the future.