On Monday of last week I had the (mostly) pleasurable experience of speaking on an IAB mobile advertising panel in New York. I say mostly because, technically, 10% of me burst into flames on the subway and never actually made it. 

Panels like these used to be the stuff of software geeks, carrier overlords and some gnarly, unholy combination of the two. The conversations were invariably technology focused as people scrambled to make sense of a fragmented industry growing at a bewildering pace. It took a few years longer than it should have, but at some point walled gardens, WAP decks and SOAP APIs gave way to use cases, ROI and sound design principles. 

The conversation has shifted again, and now people are beginning to contextualize mobile as part of a branding continuum, where digital, traditional and emerging media converge to tell a story that engages people in motion. Simon Bond, CMO of BBDO, pointed to a study his firm had done which found people are more likely to wake up and reach for their mobile device rather than their lover (or themselves, for that matter). Razorfish is doing really ground-breaking stuff in mobile advertising, working with publishers and networks to create dyanmic new ad models. Michael Collins of Joule talked about how the metrics mobile provide are on par with - even exceeding in many cases - the analytics available on the desktop web. 

At Definition 6 we are bringing serious innovation to our mobile clients. From mobile websites that predict behavior and present contextualized experiences, to mobile marketing applications that leverage unique combinations of variables like speed, distance, orientation and time of day, we are helping our clients deploy mobile marketing strategies that transform their businesses.

Check it out:
 


It begs the question - how is your firm using mobile marketing and advertising to build your brand? If you aren't, somebody else is. 

It seems that everywhere I turn, Americans are going green: Recycling cans and bottles, carpooling with friends and co-workers, and using reusable cloth bags. As someone who has carried cloth bags for over two decades, the change in mindset fascinates me.

For some it is a concern for the next generation, while others are frustrated over the rising cost of raw materials and food (remember when ice cream came in half gallons, not pints). As the world’s population approaches 7 Billion (ETA October 31, 2011) I wondered what are America’s largest corporations doing to address who they are and what they do through a lens of corporate responsibility and sustainability?

As big as annual reports were in the 80’s and 90’s, we are entering the age of the CSR. And though integrated reports may eventually replace annual reports, 52 percent of Fortune 100 Companies included statements of Corporate Social Responsibility (CSR) in their 2010 annual reports and 10-K.  The push for non-financial reporting comes from a combination of the current economic environment, public demand for transparency, the rise of social media, website integration and third-party verification.

In my two-part series on Vault.com, I discuss – and analyze - seven main trends that impact every company going forward, whether they already report on non-financial goals, or continue to put it off for a variety of reasons, but mostly, for fear.

Check out: Seven Conflicting Trends

One week later and we are still hearing great things about our after party held here last Wednesday night during the PromaxBDA conference in NYC.  Mainly, we are being asked, where are the pics?? Well, the wait is over my friends...the photos are posted!  Thanks to Kelly Samardak, our amazing photographer, we have almost 500 pics posted to our Facebook page. 

We had 20 people receive free tattoos from Daredevil Tattoo, thanks to our sponsor PointRoll, and many others having fun with some temporary D6 tattoos, as well as Henna body art (although Jeff - I'm not sure this actually counts if you just get your dog in Henna!)
 

Our other sponsor, Avid, held a raffle and gave away a FREE license for its media composer software! 


Guests also enjoyed hanging out in the Bob English Pub where we were serving Pimm's lemonade drinks and varieties of British beers like Boddington's and Newcastle.



Some may even remember seeing Gilbert Gottfried, who made an early appearance and enjoyed some great photo ops! 



And the party wouldn't be complete with a surprise appearance from the Toxic Avenger, who seemed to get his groove on the dance floor quite a bit!




It was amazing to host this party as a way to say thanks to our clients and to celebrate the amazing work we are able to produce together. We look forward to hearing your feedback - we hope you enjoy the pics!  Can't wait to see you all at our next shindig!

Last night's Definition 6 PromaxBDA after party was one of our best events to date. Special thank you to Avid and PointRoll for sponsoring our party and making this all possible.  Everyone had a great time getting inked with real tattoos, henna body art for those less brave, raffles for a free software license for Avid technology, posing for photos with the Toxic Avenger and dancing on our roof deck disco to the sounds of DJ Elle.

Here are some early pics below. Look for all the photos from our party on the Definition 6 NYC Facebook Page.

And tonight we are up for 4 BDA awards... here's hoping the party continues!

It's not every day you have the opportunity to work on something that screams cool from almost every aspect. Let me see if I can summarize this particular opportunity: vampires, witches, werewolves, Alan Ball, Anna Paquin, Alexander Skarsgard, Ryan Kwanten, Kristin Bauer, Deborah Ann Woll, HBO, Social Media, Facebook, complex Flash layering, detailed motion design, great creative, great script, personalization in full motion video, high availability, cloud farms, CDN's, open source, high performance where every millisecond in processing counts, and integration with multiple SaaS providers and their API's. 

That pretty much describes what Definition 6 recently had the opportunity to work on and launch with the hit HBO show True Blood that premieres season 4 this Sunday, June 26th. We helped them launch a Facebook application called IMMORTALIZE YOURSELF that takes a piece filmed exclusively for this purpose featuring many of the famous cast as a bridge between season 3 and season 4 and using Facebook connect you'll find you and your Facebook friends are a part of the video. Some go missing, others apply to be Fangtasia dancers, and you get to be an assassin. You're all featured in the show's famous credits and opening. And when it's all over you can re-do the whole thing but pick which friends you want to be in which roles. And while the concept of using a bespoke video with social aspects may not be entirely new or unique, I feel pretty safe in saying that the level of detail we achieved in the personalization is pretty rare. Everything looks far more a natural part of the video than almost anything done previously. A lot of love went into that effort. The same goes for attention to detail in the performance of the application for something involving so much Flash, video and detailed interactions.

Plus it's just cool. It's also pretty rare in this business to do something that achieves a 99.9% positive sentiment rating with less than 0.05% technical failures in application delivery. You can check it out for real yourself here. And you can see an example of one of the videos below, this one featuring many of the team members at Definition 6 that worked on this project.

And as an added benefit, I and many of the team members had about 45 days to immerse ourselves in the brand by watching 3 seasons of a show with brilliant dialogue, and a great story that's chock full of fantastic effects, great actors, and plenty of hot vampire sex.

Our thanks to a great client. The True Blood team wrote a great script, and all the video production work and editing was done by HBO Creative Services. As always the cast and crew of True Blood were fantastic in their performances. The marketing team at HBO continues to push the edge with great ideas and non-traditional campaigns. What they have done on True Blood previously and with this piece shows how they combine innovative thinking with creative brand authenticity.

We're lucky at Definition 6 to enjoy a long-standing relationship with HBO, spear-headed by the Managing Partner of our Post-Production division, Rob Ortiz, who has been working with HBO for over 25 years. Rob, our team at Definition 6, and the great Creative Services team at HBO, especially Becca Schader, Chris Denniston and Chris Spencer, all combined to concept, POC, flush out the idea, and drive this piece with Marketing and Interactive. It's fantastic working with people like these who continue to show they are thinking well beyond the traditional 30-second spot on how to engage fans and grow audiences in a rapidly changing landscape. True Blood is rapidly approaching 8 Million fans on Facebook, and it's pieces like this that can change how you reach and interact with those fans.

Waiting Sucks. So go check out the app. And don't miss the season premiere this Sunday on HBO, 9PM EST.

Background: It’s almost impossible to describe the smartphone market accurately without sounding overly prone to hyperbole. Worldwide, nearly half a billion smartphones will be shipped in 2011. By Christmas of 2011, one in two Americans will have one. Gartner predicts that in 2011, 85% of all handsets shipped globally will be able to access the mobile Internet.

In Q1 of 2011, Android supplanted Apple as the #1 smartphone OS in the U.S. While smartphone manufacturers jockey back and forth with one another for frontrunner status, swapping places with every latest hardware entrant, only feature phones (where the OS is proprietary firmware, and NOT a third-party development environment) are left in the starting blocks; there is little churn in the segment. In fact 2011, according to Nielsen, will be the year smartphones overtake feature phones in global mobile shipments.

For brands looking to build or maintain a mobile presence, the smartphone onslaught simply cannot be ignored. Nor can it be oversimplified. Too many American marketers think mobile is merely a stripped-down digital experience, a checkbox extension of their digital initiatives created automatically through the advent of rich smartphone browsers. The reality is that, worldwide, more people access the Internet through mobile devices than personal computers. It is the first screen, not the third. That over 95% of domestic digital marketing budgets are earmarked for non-mobile initiatives is an unpardonable lack of imagination, courage and good sense.

Smart brands and advertising agencies must consider several factors when evaluating how to create a mobile presence for smartphones. First the upside of browser-based mobile applications:  

• Write once, run anywhere: This is the siren’s call of mobile that promises cost efficiencies by keeping the time spent creating, maintaining and updating applications to a minimum through the utilization of a tool-based multi-platform design paradigm.

• Universal browser availability: A rich browsing experience is all but universal in the smartphone market. If a brand presence can be defined and made accessible through a mobile web browser, it reduces the number of headaches to resolve for platform-specific nuances as long as you stick to the lowest common denominator.

 

• Support of advanced features available in HTML5: There are some components of HTML5 that will be a boon to mobile devices. Unlike mobile browsers of old, HTML5 promises offline support. Also, video in HTML5 is standardized, which will make adding video content to websites much easier. HTML5 also supports a GeoLocation API, allowing web developers to capture critical location data.

• Layout standardization: Web browsers depend on a layout engine at the presentation layer. While there are myriad mobile browsers available, WebKit and Gecko are the layout engines used by most, including Internet Explorer, Firefox, Safari, and Chrome.

Unfortunately, there are also downsides to a browser-only approach, including:  

• Browser diversity: The most prevalent smartphone browser in existence is the Blackberry, which has its own layout engine used by precisely nobody other than RIM. At the very minimum, creating a browser-based smartphone application that works on a majority of devices will require at least two completely separate development efforts.  

• Limited HTML5 support: When Steve Jobs lambasted Adobe’s Flash in 2010, he pointed to HTML5 as the author of its death spiral. Somebody forgot to tell RIM, who supports the Opera browser while remaining ambiguous about HTML5.  

• Hardware-specific amenities: Downloadable applications are typically able to access many features baked into the hardware that a browser-based application simply cannot. From the iPhone’s popular “shake” feature to other phones cameras and GPS chips, in order to unlock the richest of features from a mobile device, you have to think thick-client.

While the debate continues, what is certain for brands is to contemplate the merits of downloads vs. browser-based applications on a case-by-case basis. Mark Donovan, comScore SVP of Mobile concludes, “with mobile media consumption on the rise, the discussion of how consumers are accessing content -- whether it is via application, browser or both -- continues to be an important factor for companies looking to invest further in their mobile brands.” In every scenario, however, it is important to consider what will not work, and that, specifically, is to think that just because an HTML-based application works on a personal computer, it is sufficient for a mobile device. Mobile users have limited and hugely variable display capabilities, awkward and difficult input mechanisms, and the mobile mindset differs vastly from the PC user’s mindset.

In future posts, I’ll talk about mobile application design principles and dissect smartphone platforms in an effort to provide color for brands contemplating including them in mobile marketing strategies.

You know, I’m a big believer that the most important thing we do is deliver work that creates measurable results for our clients.  That’s why we do this...to drive our clients towards their business goals. But when we are able to create that kind of positive impact AND be recognized by the industry for our creative achievements, well, that’s even better!

I am once again thrilled to be a part of this great company, a company that was just named one of the best places to work in Atlanta.  From Emmy nominations to Clio finalists and Webby awards, we love to share this news with our clients.  Creating great work, award-winning work, is our mantra here.  Don’t believe me?  Just take a look at this list for awards coming up over the next 30 days:

• 2 Daytime Emmy® Award nominations for Sesame Street

• Clio Award finalist for Coca-Cola’s “Happiness Truck” campaign

• 2 Webby Awards
  • “Best in Show” for Online Commercial for Coca-Cola’s “Happiness Machine”
  • “People’s Choice” for Online Commercial for Coca-Cola’s “Happiness Machine”

• National round of the ADDY® Awards for Coca-Cola’s “Happiness Machine” (after already winning a Gold and “Best in Show” ADDY® in the Atlanta competition and another Gold ADDY® at the District 7 gala earlier this year).

• 4 Promax BDA awards
  • David Foster Inspiration - Art Direction & Design: Special Events Promo - General Entertainment
  • Coca-Cola Happiness Machine - :30 - On-Air Commercial Advertisement
  • Definition 6 Journey - Self Promotion/Show Reel/Company Image
  • Spike Re-Brand Pitch - Salon de Refus

• 3 American Business Awards (Stevie Awards):
  • Coca-Cola’s “Where Will Happiness Strike Next” Campaign: Marketing Campaign of the Year – Food & Beverage
  • Mitsubishi Electric Website: Web Site or Blog Awards Categories: Manufacturing
  • Bliss Flooring Website: Web Site or Blog Awards Categories: Consumer Products – Durables

• 2 Telly Awards
  • Coca-Cola "Happiness Truck" Silver Telly
  • Coca-Cola "Happiness Truck" People's Choice Telly Bronze

It’s so exciting to see so many components of what we do for so many clients recognized.  Definition 6 is fortunate to work with such great, collaborative clients that partner with us to create great work together.  Win or lose, we truly appreciate the recognition from our colleagues and peers. 

The phrase "the more things change, the more they stay the same" has been on my mind lately. Computers have become such a part of our lives that we can’t imagine life without them. Just a few years ago it wasn’t uncommon to hear that Blackberrys called "Crack-berrys," referring to the addictive nature of having email readily accessible. Today we even have a thing called "Computer Addiction" that people can get treatment for.  The children coming of age in this era they are the most informational connected generation we have seen.  Considering the architecture changes, the changing expectations, and rate of adoption the future definitely has a more interactive and mobile look to it from a computing perspective.

I recently read where mobile devices have outsold traditional computers for the first time in the 4^th quarter of 2010. Apple has been reporting sales growth while companies like Microsoft, Intel, and other companies are reporting lower than expected returns for the same period. With certain product releases coming in 2011 the anticipation is the sales trend will only continue to grow the gap.  As we see the sales trends change and more companies trying to capitalize we look to anticipate its direction and build products viable for today’s market and tomorrows.

To anticipate the direction we can start by focusing on a brief history of the mobile devices. Consider that Microsoft was an early player in this market. Compaq iPaq’s, HP Jornada’s, and others were touted as Pocket PC’s. Toshiba had one of the first tablet PC’s I remember. It even had a built in camera but the unit was very heavy. Microsoft envisioned "smart devices" and for a while had produced marketing as such. I remember they envisioned the device could be replaced and your configuration auto-magically restored. They had great vision and they dominated the early market. But while they were an early endorser and participant in the mobile field a couple of miss-steps and lack of innovation later they were behind.

It’s arguable that widespread adoption started to change with the acceptance of the Blackberry. Users were getting email connecting in ways they really hadn’t before. It wasn’t long before next up were the expectation to be able to review attachments to email. Having the internet on a mobile device wasn’t far behind that and the expectations began to speed up.  Why? because the adoption rate improved. Users saw immediate value in the functionality of these devices. But devices had different purposes. Blackberry’s did email while pocket pc’s handled calendars and other basic functionality.  I remember at one point having so many devices I felt like a techno-nerd version of Batman. While this was going on Apple envisioned the iPhone. Apple developed the iPhone in quiet and when they released it changed the market.  The change was significant enough that the carrier they worked with to support the device was overwhelmed for a time with new customers. It seemed like overnight they met and exceeded user expectations, and made a giant leap forward. Others began to follow the trend.  

User interface expectations are certainly being affected by changing expectations. How long did companies toy with keyboards until the iPhone changed the game with the popularity of its touch interface? A touch interface for a mobile phone had not been accepted until then.  Apple tried to compensate for users comfort by adding "clicking" sounds to the iPhone. But the hardware wasn’t the only innovative aspect. They innovated software are delivery as well.

The layout of the Apps wasn’t entirely new. Icon short cuts on a desktop have been around in the Mac and Windows worlds for years but the operation or implementation around the apps was. Users were able to use the devices to quickly check what they deemed the most important things.   Another expectation is the speed that these devices are expected to operate at. Long load times are not acceptable.  In addition to load times connectivity has become a key factor as well, a key contributor to the onslaught of the battling ‘G’ advertisements and related devices.  

Delivery handled through iTunes and working directly with the Apple company remains the only way to deploy applications. With the combination of hardware, software and deployment the entire platform was innovative and users liked it.

With a great rate of adoption and renewed interest in the market other players have been working to be more competitive in this market. For example, the Droid and Microsoft’s Metro concepts are two emerging or re-emerging market competitors.  With all the various players history in some ways begins to repeat itself. As they have gained more market share and their sales increased as well technical complexities re-emerge.

We still have a familiar challenge though, remember the old Mac vs PC days? Well we are there with mobile. We see different operating systems, different carriers, lack of interoperability and different devices. Consider that Adobe’s flash won’t run on iPhones. These types of complexity have a strong feeling of déjà vu for some of us. Only now we have added the extra complexity of Different networks carry different devices and different operating systems.

The innovations in both the hardware and software will continue in the space Apple has defined for a while. We are also seeing a repeat of some of the same hardware and operating system issues that have plagued IT for years.  What is different is that the adoption rate is continuing to grow. Watching over the last several months I see more executives and other carrying tablet PCs to meetings instead of the traditional notepad.  

Innovative development on the mobile platform will remain costly in some respects. Developing for multiple operating systems and different devices presents many challenges. What’s different today is that there is more of a drive than in years past to build these solutions. There are and will be tools that enable development for devices as well as across multiple platforms. However, those tools will have limitations and it will be a challenge to truly innovate through them. While working in the native system means developing different code for the same app to work on the different systems. Architecting a solution in mobile has to take into account the various considerations. Companies have to decide if they want the expense of creating an innovative app for the mobile platform or just have an app for the platform. This has a significant cost difference especially if the app has to be deployed to multiple devices.  

As customers decide their goals and directions in the mobile space it will be important for architects to use the tools available to them. The use of design patterns and object oriented techniques will be of paramount importance going forward for the software side of solution.  Creating a scalable solution for the growing functionality needs of mobile users will be critical. Considering that the hardware of the PC has evolved at a much slower pace scalability will have additional challenges in the mobile platform.

Creating a scalable solution is more challenging with the frequent release of devices and the secrecy surrounding them. Many of the tools on a mobile device have been tools available on a traditional PC. Going forward the hardware is starting to move into truly new areas.  For example, talk of the iPhone 5 and the capability of it having Near Field Communication capabilities have been going on for months. Talk has already started about functionality of the iPhone 6. Architects will be able to help customers prepare for not only the next deployment but the one after that.

The mobile environment is a market that companies cannot continue to avoid as it has passed the tipping point of adoption. But those same companies have to realize where mobile is in it maturity. Companies will not be able to build an app, deploy it, and then forget it. These apps are living in an every changing world and will need maintenance to continue operating effectively.  The architectures supporting the apps and contained within the apps must be able to scale to meet these needs.

The mobile environment is changing frequently and stepping forward in leaps we haven’t seen in a while. Developing solutions for customers means considering all the factors and leading them to understand the environment. Bobby Knight is probably as polarizing a figure in college basketball as there is. For all the negative about him later in his career, he is regarded as a great teacher of the game. It’s one of those lessons that really apply here. He said, I am paraphrasing, "we have to focus, by focusing it allows us to notice trends, recognizing trends allows us to anticipate, and that leads to action."

 

 I’m just back from spending a week in L.A. for the 2011 Spring Digital Hollywood Conference…my first time attending the show.   So many panels, fascinating people, and great topics of conversation.  The biggest crisis that seems to be facing this industry is one of identity…when I can get content anywhere on any device any time I want it, can I still say I am in the TV business? 

Having been raised on the digital side of this equation, I found it fascinating to see what an internal struggle this issue has become.  In my mind, production of good content continues to be the real source of value.  Good content will usually rise to the top.  Technology has simply increased the number of opportunities to consume that content, as well as the number of opportunities to monetize that content.  It also has introduced measurement into the equation.  Measurement puts pressure on the content providers to prove they are delivering the audience advertisers want and need.  You also hear a lot more about how targeting is so important…why waste advertising dollars on the people who aren’t buying or will never buy your product?  Well, in my opinion, reach still matters because everyone has the opportunity to be an influencer.  Whether or not you buy my products, you can still help influence someone else who does. 

One of the biggest things I took away from the event was a quote from Adam Carolla, who noted following one panelist’s description on the state of the industry, “Do we really need ‘wild, wild west?’ Doesn’t one ‘wild’ really encapsulate everything we want to say?”  The second biggest take-away was a quote from Stan Kurnit of AdKeeper and founder of About.com -  He said, “It’s not that consumers don’t like advertising, they just don’t like how we are doing it.”  With the changes in technology and content distribution, consumers have choice. They have control.  That doesn’t mean they don’t want or even sometimes enjoy advertising…it just means we, as advertisers, need to do a better job at creating ads that add value, that entertain, and that truly inform.  It’s easy to say consumers aren’t impacted by advertising anymore, but that is far from the truth.  They just aren’t impacted by bad advertising anymore.  The expectation has been raised significantly.  There are plenty of interesting examples of companies doing it right now.  But the industry definitely continues to evolve at the intersection of content and context.

Why advertising continues to fail is becoming clearer to me every day, and after every conversation I have with a client. 
 
It isn't that people are watching less TV (because viewership is up), or that we are reading less content (please, with the amount of social content, blogs, text messages, etc. - we are reading more than ever), and it is not even that consumers led by the Millennials are jaded and immune to ad messages.  To me, it is simple - advertising is failing because it focuses too much on the "big idea" and not enough on the "big connection" (You could call also it "engagement" or "emotional connection").
 
Big ideas in this context are the big "ad" ideas, not the big idea that makes a moment.  In fact, if advertising were to focus on "big ideas that made an impact," then it would be doing a much better job than it is today. But it would still be failing.  What consumers crave - and brands must deliver - is the idea that makes the connection to the way we live (and want to live) our everyday lives. 

I'm not talking merely about making sure that there is channel consistency in messaging, or even in experience.  Both are essential as I've discussed before. And I'm certainly not talking about making sure you are mobile and, if you are really smart, ensuring you have some social gaming elements in your arsenal.  Again, those are important, essential for consumer consumption and desire.  What I really mean is this.
 
Make my life better or I don't care about you!
 
Seems pretty simple to me and, to that end, advertising needs to understand that its role is fading and won't come back.  Marketing, the entire discipline, is now tasked to take the lead.  I'm talking about the four P's and the four C's too!

Our clients, friends in the business, the CEOs that we spend our days with don't want ad campaigns or advertising ideas. They want business driven marketing support.  They ask for it differently; "Our customer service experience in-store doesn't align with our brand promise in our advertising" or "Our product teams aren't listening fast enough to our consumers to make a meaningful impact this year" or "Why do I need an iPad strategy? I've barely finished executing our social strategy? Do I need an international activation team too?"
 
What I'm hearing is don't bring me advertising ideas, bring me business driven marketing ideas.  I think they are right.  Anyone else feeling like their clients are asking for the same thing these days?

Designing systems, interfaces, back end engines, databases and other system related items takes a blend of creative and technical acumen. IT professionals love to solve problems with their blended skill sets. When an algorithm is developed we sometimes even share those ideas and algorithms so it can help other systems. One case in point is my colleague Jon Taylor’s recent blog about the Observer pattern and .net event models. (Check it out of you haven’t yet-great information.) Design patterns in themselves are previous solutions that give us a solid base to develop a new algorithm from. But how many times and in how many ways have we overdesigned something?

I remember a bad decision I made at the beginning of my career in overdesign. A customer wanted a backward compatible query building wizard for their staff to use that had little technical knowledge or database understanding. However, the trick was the ‘management’ staff consistently wanted to ‘adjust’ database table and field names. The data in them wouldn’t change but the name needed to change. They were adamant about this. In coming up with a design we looked at using SQL Server ID numbers to store. This way we could recreate the same query with new names and we wouldn’t have backward compatibility issues. When built it worked great everyone had what they needed. Until a new version of SQL Server came out that instead of updating table names dropped them and re-created them. Slap the forehead time. I didn’t consider the impact or fragility change would have on the system. I focused on one direction to much.

Instead of looking to a simpler solution I went to a more complex one, or better said as an overdesigned one. I have never forgotten that lesson and I find all kinds of ways to apply it. Over the years this has turned into a philosophy and phrase ‘evolution not revolution’. Revolution is not limited to overdesigning the back end system it could be overdesigning any part of the system. In this context the concept of revolution for change is a bad one. Some of the aspects of revolution I am referring to are the destructive nature, the amount of change that has be absorbed, what can’t be changed back and the time to adjust to the changes. When your users are feeling the impact of a system revolution is this a good thing? What causes them to feel that?

 We aren’t talking about users feelings at the deployment of a system. We are talking about the users feelings over time. One example, I heard company say they couldn’t adjust the size of a banner frame for their customer because they had built it custom. It would take to many hours to change. That small issue was the start for the customer losing faith in the product and the team that created it. Eventually that team lost the business. Why? They created a revolutionary new banner frame that was great. But when the customer needed change they couldn’t accommodate efficiently and cost effectively.

When architecting we have to be able to break down a system to its critical components and work forward. We have to keep in mind that the parts of our systems need to evolve and grow. To best serve our customers we have to be diligent in evaluating if our design is evolutionary or revolutionary. Thinking of the quote ‘an ounce of prevention is worth a pound of cure’ what are some simple ways to prevent overdesign. Collaborate; working alone in a vacuum is not always a great way to a solution. One of the reasons collaboration is such a great tool for teams is someone is always looking for a simpler way. If you aren’t working with a team or before you review with the team, ask yourself am I designing to the requirements or am I trying to add value. Sometimes in trying to add value we get so excited about what the system could do we forget that the customer may have never wanted that.

Prevention begins early in the process. Understanding the workflows and business processes associated with a system is critical. Architects, testers, developers, and customers all need to be on the same page. In gathering requirements we have to dig to the deeper process to design a quality system. With an understanding of the business processes, the entire development process begins from requirements on down reflect the beginning is a base.

Realize that for many systems that have heavy interaction, users need time with the system. This is important because I am frequently surprised with the direction a customer will drive a solution over time. It is so important to consider this during design as well as critical to take the time to design. It surprises me that even today how many developers and team almost go straight to code.  Take the time to focus on the base and look for evolution. For architects and developers design patterns are great if it fits and gives your customer what they need. Consider that the only constant is change. Do you really need that whiz bang new pattern or will something simpler work better?  

Create a system that can evolve with the customers’ needs over time not revolutionize today forgetting about tomorrow.

A common refrain in IT is to "begin with the end in mind." It is one of those refrains that has been repeated so many times that it quickly becomes ignored. For systems that collect data, "the end" may be very complicated. Some data collected can be presented in reports and that is sufficient. Reporting is such an easily understood term and used so frequently it often hides complexity. It’s in that complexity that frustration for system engineers and customers alike begins.  Because of that, complexity reporting is often not a cost effective means for exploring data. Complex reports can take a significant amount of time, (translate that to cost), in order to complete.  When users are looking at the data to see what trends or interesting anomalies appear, reports just aren’t efficient. We begin by recognizing that "the end" can be as complex a system as the front end collection system.

Architecting an analysis system can comprise many parts ans some of the most overlooked and underutilized are OLAP tools (Online Analytical Processing). As such, optimizing those tools and working with users to capture the areas the tool needs to work with are important.  It’s that OLAP part of an analysis system that we begin to explore in a bit more detail.

So to begin working with OLAP, two terms will immediately need to be understood--dimension and measure. Working off of a graph diagram is the best explanation to start with. Looking at Figure 1, notice the X–axis of the graph. In OLAP, that axis will be called a "dimension". The calculation that occurs at each notch in the access is called a "measure".  That in mind, figure 1 shows location as the dimension and debt amount as the measure.

 

Figure 1.

Using the same basic graph model, imagine that we add a new dimension debt type. So now the intersection that points between the dimensions debt type and location show our debt amount measure. The great thing about OLAP tools is that depending on the particular tool, they can look at many measures at the same time, can flatten a dimension out, and look at all of one and a particular point or notch on another dimension. So, why are they called cubes?

 Figure 2 shows the addition of the Calendar dimension, essentially time limited to the granularity level of day. With that dimension our graphical representation now resembles a cube, hence the name. While graphically we can only represent 3 dimensions clearly and easily, an OLAP system can represent hundreds of dimensions.

  

Figure 2.

OLAP tools provide different types of interfaces that allow users to simply drag, drop and click to explore data. Some tools like SAS Enterprise Guide can even graph dynamically while the user is exploring. Robust tools like this provide a good user experience and enhances the analysis system. Many systems like SQL Analysis Services connects directly to SQL Reporting services, allowing for reports to be built quickly and easily in addition to the normal canned reports. So, what does an architect need to consider with a basic understanding of what OLAP does for users?

An OLAP system basically represents hundreds or thousands, depending on the number of dimensions and measures, of queries working against the data simultaneously.  In processing, systems like SQL Analysis server offer different processing options to do as many pre-calculations as possible. These processing options are HOLAP, MOLAP, and ROLAP, but let’s not get into the weeds. What we need to know is that the system is going to perform some level of pre-processing and store that pre-processing. To do that pre-processing, and for that matter, processing, there are data structure designs that work more efficiently with OLAP. Some will refer to these models as Star and Snowflake patterns. What architects need to look at is that the data model for the collection may be differently optimized. Don’t get into discussions of normalization here, as it doesn’t apply. Both the data models can be normalized but still not be optimized for their working intent. Normalization does equal optimization for all cases.

Looking at figure 3, the data model shows a table labeled with the prefix "fact." The fact table in an OLAP model is where the values to calculate measures are usually stored. Keep in mind that a measure is an individual calculation and the facts are the values that enable those calculations. To help the OLAP system work, a group of foreign keys relate out to the tables with the prefix "dim." The dim tables may be single tables as represented with the debt type. Or they may be hierarchal as represented with the Calendar dimension.

Figure 3.

What creating a database model like this allows for is the OLAP engine to process quicker. Creating this model may result in ETL routines being created, which is a benefit, as dimension data can be transformed from being collected one way to another in the OLAP data model. This provides even more flexibility towards what we can represent, as well as interjecting new considerations. How often do we move the data? Do we copy the data or do we move it? An analysis system has to consider all of these options.

OLAP can also be a data source for the front end interface and provide new dashboards and functionality to the collection interface as well. Visual Studio.net for example, has quite a few capabilities in this area that will be in another blog coming soon. Getting back, this allows analysis systems the capability of being an integrated system in new ways with the collection system.

If you see the value of OLAP, look at it as another tool for the analysis system. Yes, it can be a replacement for some reports. However, a solution will still include canned reports, as we are always going to need those.  A full analysis system is going to have more components to it and be fully thought out. With this end in mind, if you were to design the exploration, reporting, queries, and analysis you want a system to provide, how different would the collection system be?

Many systems are designed from the point of view of the collection system. After all, that’s what most users will see and work with. So we focus there. Only later to realize we want to analyze data and we have the wrong level of granularity or we are missing a variable. Many know that the cost and time increase significantly when changing a constructed system versus at design time. Have you considered what the cost is of building the systems in the wrong order? Is the tail wagging the dog?

Recently I spoke at SXSW Interactive regarding presentations for right versus left brained audiences. I was asked about the ideal client and I brought up a past experience that I only recently discovered again. As anyone working in creative communicatons realizes, few logos and identity projects ever survive to maturity, just as most businesses fail in their first few years.

My ideal client at the time was Selima Salaun of Selima Optiques. I first met Selima in 1986 at the Alain Mikli Optique on 5th Avenue. I was a fan of Mikli eyeglass frame designs and owned several pairs of them. Selima was a creative spirit and a skilled designer and hat maker in Paris before opening the New York location for Mikli. We became friends as we were both fans of opera.

A few years passed and I received a call from Selima, who told me she was working on opening her own designer line of eyewear and had located a store space in Soho for her location. She had been thinking about her business name and the importance of the eye in her business, so she was calling to see if I could design a visual icon that would include both aspects for her new business: Selima Optique. I was living in Roswell at the time, so we would review over the phone and through digital files I sent. In the end, the right design came about from using a hand-drawn technique. It was distinctly different from other brands in that space at the time. Selima accepted the final comp and files, paid me and I lost touch with her over time.



Over 15 years passed, and then I rediscovered that Selima Optique was not only still in business but had expanded to Barney's, J.Crew and the Selima Optique brand was both trendy and global. Not being a celebrity or fashion follower, I never had an idea that she was doing custom designed shades for Bono, Lenny Kravitz, Michael Jackson and others. But I cannot think of a more deserving person for this success.

So what makes a great creative client relationship? The client with a passion for their brand or vision goes a long way towards good work. And they must also really have the authority or ownership of the work. I have been involved in too many failure stories where the chief marketing officer took it upon themselves without much involvement from the ultimate owner or visionary (typically a chief executive officer). The result, no matter the level of the work, was not well received and died a quick death.

Another other most importance factor in success with brand work is having the courage to trust in the team you have engaged to build the work. Success comes to those that dare go the opposite direction of the competition. It requires a lot of bravery and trust and that is not common in general, let alone in building a significant brand.  The western corporate world is littered with “curving stroke logos” that frankly all owe their inspiration to the Nike mark. The original swoosh was created by a single designer working directly with a right-brained visionary.

This past week I had the opportunity to present at the Business Development Institute (BDI) Social Consumer event in New York City and speak about one of Definition 6’s great customer case studies and success stories. The event was attended by over 300 people that work in marketing and media and come from a great variety of big companies including MTV, McGraw Hill, American Express, Citi Group and many others. I’m pretty sure I was one of very few people in the room that works mostly on the tech side of things.

It was a great event. The speaker from Unilever, Senior Communications Marketing Manager Stacie Bright (@Dove), has me honestly considering trying out Dove Body Wash for Men, and Senior Brand Manager Juliet Wilson from Kotex and Organic’s Julie Lee (@julielee75) showed off some truly game-changing work they are doing and even had the entire audience in various stages of discomfort reciting the word “vagina”. But the most energetic speaker by far was Bonin Bough (@boughb), the Global Director of Social Media for Pepsico who delivered a fantastic keynote speech and showed off some very innovative work they’ve done while talking about the importance of having Digital Fitness throughout an organization.

Of course, this made it a bit ironic that my presentation, which followed Bonin’s, essentially focused on some of the great work we’ve done over the past year or so with our customer Coca-Cola. So what’s a tech geek from Atlanta whose company does work for Coke supposed to do in this situation? Well… mostly just try to do his best to represent and share a little Happiness. You can view my presentation which is about 20 minutes long and judge for yourself. I think it went pretty well with the exception of a little technical glitch at the end where the presentation laptop froze up, though I think that happened because I was making jokes about IBM’s Watson at the time. Enjoy.

Change sucks. I’ve heard the glib rhetoric and seen the shiny happy bumper stickers proclaiming the staunch opposite. They're often paired with an equally shiny sticker of a cat skeleton with a smiley face stuck to the front of it. "Change is good," but apparently not for the cat.

Now, I’ll admit, for some things, change is a genuinely welcome event – underwear, socks, cat litter, and most immediately my empty pint glass come to mind. However, with web application development change sucks! Change means long nights rolling in new code to replace old. Change means endless cycles refactoring someone else’s code, often on shoestring documentation and littered with dependencies you don’t know about until you trip over them. How often have you fixed one bug and 3 more pop up? Poorly written code, like it or not, is a fact of life.

Blame it on too many requirements in not enough time, or whatever you want to, but the sad truth is most developers build what they have to without thought of what it takes to support it once it’s running. If it’s your task to have to go back in and add a new feature or change existing functionality, yeah, change sucks. And the number one culprit in contributing to code sucky-ness is dependency. Updating heavily dependent code is like planting corn in a minefield, you might get a few new stalks in ok, but it’s only a matter of time before something’s going to blow up.

So, how can we, as developers, do eachother right and make sure the code we write won’t blow somebody’s refactoring effort to smithereens? Especially if it might actually be us doing the refactoring? Limit dependency in our own code! Right! Ok, so how exactly do we do go about building projects that aren’t refactoring minefileds? Glad you asked.

The easiest way to reduce dependency between two pieces of code is through a façade. Let’s go with an example. Take a look at the code in Segment 1.

Code Segment 1.

    class Example

    {

        //Default constructor

        public Example()

        {

 

        }   

        //Public method

        public void DoSomething(String thing)

        {

            //process thing

        }

    }

    public class Caller

    {

        String stuff = "stuff that needs to be done";

        Example xmpl = new Example();

        xmpl.DoSomething(stuff);

    }

 

Pretty standard stuff, right? A simple Caller class that instantiates a version of our Example class, then calls the Example class’ DoSomething() method. Pretty standard, you’ve probably written hundreds of similar code sets. But what if we had to update DoSomething()’s code to include some new functionality once it had been pushed into production? We’d have to change both our Example class, our Caller class, and anybody else that was dependent on Example to do something. Total pain.

Now, there are lots of ways we can reduce our caller class dependency in our example class. We could setup a factory to handle the instantiation of the example class, throw in some dependency injection, or heck, even define an interface we could program all our caller classes to [Love interfaces. Love them. Separate declaration from implementation in one easy step. But sadly, it’s another post…JT], but our intent here is relate to you just how easy it is to separate the declaration of our DoSomething() method from its processing through a simple façade, and in so doing significantly reduce the dependency any class implementing our Example class has to deal with. Let’s get to it.

What we’re going to do is use a really easy implementation of the Façade Pattern which Microsoft itself uses extensively throughout its .NET framework – a Non-Virtual Interface (NVI).

Oooo. Aaaahh.

Ok, big name for really simple solution to break dependency between entities. Basically what an NVI is a public method declaration that calls a private internal method to carry out its processing. I know, sounds too easy right? Right! It is! But what it does for us is cleanly separate declaration from implementation.

Oooo. Aaaahh.

Ok, I’ll stop, but this is wicked cool stuff. Check out Segment 2

Code Segment 2

    class Example

    {

        //Default constructor

        public Example()

        {

 

        }   

        //Public method

        public void DoSomething(String something)

        {

            DoSomethingElse(something);

        }

        //Private method to do processing

        public void DoSomethingElse(String somethingElse)

{

            //process something else

}

    }

 

    public class Caller

    {

        String stuff = "stuff that needs to be done";

        Example xmpl = new Example();

        xmpl.DoSomething(stuff);

    }

 

Again, it looks too easy, doesn’t it? In fact, if you don’t stop to think about what this code is doing for us, you’ll miss it, it’s that easy. Say we had to support some new corporate mandate in our DoSomething() method once it had already been released? [We have to log everytime someone does something! Log it! Log it I say!] Well thanks to our tidy NVI, all we have to do is go into our code, and add the new logging functionality to the private DoSomethingElse() method, and we’re done. Done! Our Caller class doesn’t have to be touched because its method call into DoSomething() hasn’t changed!

It’s starting to make sense, right?

So, exposing a public interface which calls a private implementation is the core of what a Façade Pattern is, and a Non-Virtual Interface is a neat trick you can use to keep your own code separated and dependency-Free.

Oooo. Aaaahh.

Sorry, couldn’t resist. Imagine how much easier your refactoring efforts would be if all your existing projects implemented NVIs throughout their codebase? Would certainly reduce the number of senseless refactoring minefield injuries, wouldn’t it? Oh yes it would.

Life might acutally be pretty good. Change would still definitely suck though.

Web application development is on the precipice of a revolution; through a confluence of independent practices and technologies, the way in which web applications are built and consumed is about to change radically.

What is happening is a fundamental shift away from the current state-heavy, page-based processing model to a loose collection of lightweight web-based services that an empowered client can consume to provide full functionality in a composite fashion. It started with AJAX, grew substantially with MVC, took shape with REST, and is now about to break wide open.

Web development up to this point has been tied very closely to processing HttpRequests on a page-by-page level. Remember classic ASP and ColdFusion with code intermixed with html on the page? [Honestly, was there ever a worse language to work with than VBScript? If…EndIf…Next? Please. Microsoft still owes the development community a sincere apology for making us suffer through the miserable excuse for a language that was VBScript – JT] Nightmare! ASP.NET brought the separation of code and markup, which was a huge advancement, but brought along with it a heavy page-processing model and cumbersome state management tied to the server, or worse, hashed and embedded in the page itself. [He’s using ViewState! Grab your torch and pitchforks! Raah!]

AJAX was the emancipation proclamation that freed web development from the confines of postback processing. Utilizing the substantially overlooked XMLHttpRequest object [Thank you Microsoft for developing the XmlHttpRequest in 2000 but not incorporating it into a browser until 2006], developers were suddenly free to call back into their server for processing without having to post an entire page’s content and wait for it to come back. In its entirety. Which took time. Like reading this. OK, you get the point. AJAX did more to enliven the web than any plugin had ever done previously, [sounding the death knell for Flash and similar technologies, R.I.P] and was the timely injection of functionality and responsiveness the web desperately needed. AJAX, however, despite its freedom from the page processing lifecycle, is still inherently tied to a page – its whole intent was to be able to refresh parts of a page without having to throw the whole page back to the server, so no matter how you implement it, Ajax is fundamentally tied to a page. Figure 1 is a good representation of the current state of web development, with different clients requiring dedicated applications to process information over the web

Figure 1 - The current state of web development


MVC changes how we even think about web processing by changing the url from pointing to a physical file sitting in a directory on a server somewhere to a call to a method in the code of our application. This is a huge change; the url doesn’t point to another page or image or even another site, it fires off functionality on demand.

Now, combine this url-based functionality with a robust client-side library such as jQuery, which is capable of consuming and populating services of various types, and you have the basis for a new processing model for web applications. Consider, free from the duty of page DOM processing, jQuery can intercept a click on the page and call any service we want instead. In this scenario, jQuery acts as a controller of our base page, synchronizing calls to a myriad of distributed lightweight service endpoints to deliver full functionality as a composite set. Incorporating the capabilities of HTML5, we’re free from supporting multiple plug-ins to support media, which we can also stream in from services as required.

So what’s really happening here is a combination of the extensibility of service oriented architecture (SOA) with AJAX-style web client processing for the purposes of delivering web applications as outlined in Figure 2.
 
Figure 2 - Composite Web Application Development



This model offers significant benefits over previous web development techniques including, but not limited to:

·         Extensibility: need a new feature? Build a new service and .js file to consume it. You’re done. And you can implement both to the live app without disruption of service.

·         Scalability: services can be provisioned collectively or separately to address performance concerns – which fits perfectly with modern virtualization practices.

·         Performance: by relieving the server from redundant page postback processing, and moving a large portion of processing onto the client, the server is free to dedicate more resources to handling dedicated lightweight service calls, significantly speeding overall responsiveness.

·         Maintainability: encapsulating functionality into independent services means if changes or added features are required by the application, only one service needs to be updated, not the entire application. And as it’s the service that will be altered in most cases, production environments will not have to be impacted by the service refresh.

·         Client Agnostic: Implementing this model frees processing from the page, meaning the same services which feed our web application can easily be consumed by a mobile or platform-specific client, extending the development effort in a parallel fashion.

Service based applications have been highly regarded for quite some time due to their clear separation of concerns and inherent benefits of such an implementation, but the challenge thus far in utilizing a Service Oriented Architecture with web applications has been the limitations of the page/postback processing model. Combining the capabilities of url-based method calls with AJAX-style lightweight service consumption, a composite web application model is emerging which will change the way web application development is considered and pursued. The revolution is coming.

Did you just get home from SXSW? If you burned the candle at both ends and got the most out of SXSW, then you left Austin with a slew of new connections, strengthened existing relationships and had a lot of fun. What a marathon. Panels and meetings all day, networking and parties all night, rinse, repeat. It’s exhausting, but it’s worth it because of the connections you’re building. Where most conferences focus on big product launches, SXSW is about the personal connections and networking. To quote something Annette Shade tweeted during the 2011 South By Southwest Interactive conference, “CES is to products what SXSW is to relationships.” She’s right. You should to move quickly, and within the next few days you should to organize your new connections and create a solid foundation to enable these valuable new relationships to grow.


Original pioneers of NBC News’ social media presence; Jon Accarrino, Jim Long and Frank Radice

1) Organize Your New Contacts
You probably came home with a sizeable stack of business cards even if you use a digital networking app like Addieu or Hashable. If you haven’t heard of these apps yet, don’t worry, I’ll cover that later in the article. Right now just group all your contacts and cards, physical and virtual, into the following buckets:


Bucket #1: Business Opps
•    Business Opportunities
•    Sales Leads
•    Potential Business Partners
•    People You’d Want to Work With or Hire

Bucket #2: Influencers
•    Press
•    Bloggers, Super Socials & Podcasters
•    Industry Mentors

Bucket #3: Expanding Your Network
•    Competitors
•    Counterparts
•    Colorful People to Have at Parties

Bucket #4: Trash
•    People who scare you
•    Criminals
•    Jerks


2) Take Out The Trash
Did you get a business card from someone who threw up in the hotel elevator, sprayed business cards around the convention floor like a lawn sprinkler, proclaimed themselves to be a “Social Media Guru” or just came to the conference for the free food and drinks? You’re probably not going to do business with these people. Toss their physical business cards in the trash and don’t worry about any of the digital connections you made using apps like Hashable. They will fade away with time.



3) Scribble Some Notes
Take some notes while everything is still fresh in your mind. That will help prevent you from finding that “mystery business card” that you don’t remember on your desk in a few months. If you are a “paper person,” then grab a pen, flip over any business cards and add write a few notes on the back like: met at CNN Grill at #SXSW 2011, her favorite beer = Fireman’s 4, had dinner at Vince Young Steakhouse, etc...

If you’re a “digital person,” then add everyone to your address book and write these notes in the notes field.

4) Carpe Diem
Did you leave SXSW with a hot lead or press opportunity? If you feel you have a real and immediate business opportunity, then act on it. A lot of times, people come to conferences looking to do business pretty quickly. Seize the moment, just don’t get overzealous. There’s nothing more annoying than having your entire team harass someone you just met with phone calls and emails. That’s a sure way to sever a relationship. Assign either yourself (ideal) or one person in your organization to follow up.

5) Socially Connect
For potential or future business opportunities, you should take a much softer approach. Social media can be a powerful business tool when used properly.

Follow everyone in Buckets 1, 2 and 3 on Twitter and send a connect request on LinkedIn. Then, use a grouping solution like Twitter Lists or Hootsuite to create a listening dashboard. Create one big SXSW list with everyone or break it down into smaller groups like the sub bullets listed in the bucket above.

Now that you are listening, it’s time to say something. Maybe follow up with a quick tweet. You both just attended a conference and have something in common to talk about. Not sure what to say? You know all those pictures and video you took on your camera phone?  Every piece of media can be a valuable conversation starter and help you strengthen a relationship. Email or tweet your top SXSW connections a photo from an event you both attended, a link to something that you both talked about, or a relevant SXSW blog post... Like this one. :) For the next few months, monitor and engage your new connections when appropriate.

You might have noticed that I left out Facebook. Do not send a friend request to a business prospect on Facebook. Although some people use Facebook for networking, that’s not really what it’s meant for. Facebook is for real “friends.” Unless you know the name of their family dog or have been on a double date with the person, then you probably aren’t really “friends.” A Facebook friend request for someone you barely know can actually be a little creepy. When in doubt, wait. Start with LinkedIn and Twitter and send that Facebook friend request when your relationship is a little stronger and it feels right.


6) Go Paperless Next Time
Do you have a smartphone? If you haven’t already, go download a professional networking app like Hashable or Addieu.

Addieu will quickly exchange digital business cards with someone and automatically follow the person you are connecting with on a variety of platforms.

Hashable will exchange cards and follow new contatcs like Addieu, but Hashable will also track your connections and automatically build out your personal network over time. It’s my personal favorite. Every time you meet someone you want to “connect” with, you just select what type of meeting it is (#lunch, #justmet, #beers, etc.) and type in either their name, twitter handle, or email address.



Hashable will even check you in on Foursquare or post a tweet about your meeting if you want. As a result, you get a next generation address book that not only organizes contact info but the engagement and strength of that relationship. Plus less business cards means less data entry work for you, and it even saves some trees.





So there you have it, 6 ways to get all your contacts organized so they can grow. Hope you found this blog post useful. Let’s grab #beers next year at SXSW! @Definition6’s treat!

Many application development companies regadless of web development or windows development want or need to implement search functionality. However, it is a commonly underestimated function and it continues to evolve over time. Interestingly users want search to have minimal to no interaction while having a maximum result. With data and collection systems becoming more and more complex this becomes and increasingly difficult challenge. I remember a system I was architecting for a customer where the customer wanted to enter a DNA result that consisted of an 800 to 1600 character string into a web application and have it search a database using an algorithm providing scored search results. The customer was convinced that a basic desktop machine would act as a server and be able to conduct the search against a large database efficiently. The production architecture needed to support the customers’ performance requirements was a High Performance Computing hardware environment.  Like many customers they didn’t understand the complexity of certain functions. Thinking through this topic recently had me researching how functions in systems and their architectures evolved.

Architecting a system today has many facets, and search certainly is a prominent one. Searching for information is not a new concept but a heavily evolving one. Once computers evolved beyond just basic mathematics and started capturing, storing and manipulating other data the need for search began. Early systems collected data that was somewhat structured in files and databases. Search functions found data quickly within those structures. With the development of relational databases and more complex data capture search the tools for search had to grow. Also the acceptance and use of computers was growing and more and more.
Architecting search within a system has consistently had to recognize simultaneous evolutions. Database tools added the ability to index tables to help search perform better. Search appliances like Wizards emerged for more technically savvy users to pull data from a data source. Multiple levels of searching complexity were emerging. While these searches largely dealt with structured data stored in systems, at the same time this evolution was occurring what cannot be ignored is the emergence of the internet and its impact on search. 

Early on companies like Yahoo profited on the simple concept of locating content. While this wasn’t structure data as in databases internet standards of things like meta-tag’s and other items made it possible for users to find content early on. Searching on the internet allowed users to enter terms and content related to those terms would be returned. Later companies like Google would improve the algorithms and set that industries standard for a time. E-commerce companies were also integrating user shopper experiences with search as a means of driving revenue. So while a user shopped for shoes, related items and previous shopping items would appear in the links and advertising throughout the system. While the motives were different the capture of information and providing relevant data back is essentially an implied search. The evolution of the internet and its potential was impacting local systems.

Users’ expectations were changing as the interaction was to enter in a few terms and that brought back content they wanted to see. At the same time computers continued advancing in hardware and use. Pictures, Videos, art, music files evolved to become more common to be stored on systems. In fact digital has become so big that companies like Kodak have stopped producing film based cameras. Users have embraced and ran with the lower cost and portability of digital media. This new media has presented a new challenge and forced search to evolve in multiple ways again.

Architects and systems were faced with growing use for search.  Users were searching as an exploratory exercise as more complex data and more types of data were being captured. Allowing for the advancement of tools like Online Analytical Processing (OLAP) and reporting tools. Users weren’t looking for specific data as much as looking to see what trends might appear in the data. These tools while technically complex have easy to use interfaces that allow users to review and analyze data. The complexity lies in the architecture and backend. The emergence and development of these tools was a move from appliance parts of a system to search to a full blown system of its own.

Users now expect applications to be able to search both structured and unstructured data. They want to give as little information as possible and quickly find very relevant search results. Algorithms and techniques for searching continue to advance because they must--including incorporating e-commerce like changes in the system and having subtle changes help the customer get to the results they want more quickly. One of the many reasons unstructured data evolved was not only digital media but mobile devices.

This latest evolution has occurred simultaneously with the acceptance of mobile devices. Now users have a high level of portability and connectivity to data. These mobile tools work quickly using touch screen technology and other key changes that impact the user experience for working with data. This has resulted in a need for better performance and system architectures that incorporate different devices, connectivity, and desired results.

Today’s cutting edge searches involve grabbing information from a part of a picture and searching for related information. Searches that work from audio files or live audio and provide related information quickly on portable devices is another technology that has been developed. Users want more with less required of them, resulting in more complex algorithms and models for searching.

Successfully architecting a system means taking a lot of factors into consideration. A successful solution can't overlook what the implementation's search functionality has within an enterprise system. Architecting search as a part of a system today means taking many factors into account. Understanding the user’s expectations and desired results has become critical to the successful use of a system. What devices are targeted for use, what is the complexity of the data, what type of data, and other questions like these are all key to get answered to develop a successful search system. Working with customers to identify the business rules that lead to implicit and explicit searches is important as systems more and more are expected to show relevant data.

The other day I was discussing the finer points of flat file conversion in Application Development (or lack thereof) with one of my colleagues on the Definition 6 Architecture team, and we got into a rather interesting conversation regarding how much of today’s data still gets transmitted via data documents of some kind. Modern programming practice is to transfer data via on-demand services, it's faster, easier, and in most cases much more secure. The truth of the world we live in is that most older systems, and an uncomfortably large number of new systems still  transmit data to business partners across the world using plain old text files filled with data in whatever standard they were supporting when the system was built. How's that for communcation planning? [Note: this practice has a name – Electronic Data Interchange, or EDI, and Microsoft has an exceptional tool for optimizing and synhcronizing these efforts, Biztalk 2010, but that's another post entirely – JT]

Now, my colleague, who shall remain nameless, pointed out that these systems have been functioning without problem for years, so something must be said for the persistence of such file transfer processes. My carefully worded retort was that this perceived stability was the direct result of poor sods like myself working my tail off to keep them going. At this point I feel it is my duty to disclose that while it doesn’t seem to have affected him personally, my colleague was once a long-time employee for an organization many would lay the finger of blame at for the creation and continued use of this file-based data transfer silliness, even in the face of more reliable, efficient alternatives (I won’t disclose the name of this much maligned organization, but it rhymes closely with "aye, be them"). And conceding to my colleagues’ point, this is surely the reasoning why these processes still exist today. (A clearer case of ‘if it ain’t broke, don’t fix it’ you’ll never find)

Thus, like them or not, we as developers are stuck with these processes, and it is up to us to implement solutions that accommodate them as best as possible. Now, a simply staggering majority of these systems rely on FTP transfer to distribute their data files, and typically it is up to the receiving party to pick up these files from an ftp "dropbox" and process them in a timely manner. FTP transfer has its own peculiarities that do not help this process, however, the most prominent of which being unpredictable transfer times and the disparity between the file arriving at the destination dropbox and the file completely downloading in its entirety from its origin. Aha! Herein lays our opportunity.

The problem of uncertain FTP delivery schedules causes file recipients to either delay retrieving the files from the dropbox until a time when they are certain the file will be there, "nightly file transfers," etc., or resort to "polling" to periodically spin up a process to look for the file being in the dropbox, and if it is, begin processing of the received file or pass it along to another process for further manipulation. These practices are inherently flawed and incur extensive overhead in time and system resources while constantly checking to see if the file has arrived, then locking resouceswhile the file finishes downloading before the processing of the files data can finally take place.

With .Ne however, these problems can be solved easily enough – if we can’t change the process, at least we can make it better, right? Let’s get started.

The key to our solution is a little-known member of the .NET System.IO namespace, the FileSystemWatcher class. The FileSystemWatcher does exactly what its name implies – it’s a lightweight object that monitors a directory and raises events to any changes that occur. We’re going to build a small console application and use a FileSystemWatcher to monitor our ftp directory for any new files that get created. Code Segment 1 details our System.IO.FileSystemWatcher implementation.

Code Segment 1

class Program

    {

        static void Main(string[] args)

        {

            //This should be an actual ftp directory path,

     //preferably from an App.Config file      

            String ftpPath = "path-to-ftp-directory";

 

            //Our watcher! 

            FileSystemWatcher watcher = new FileSystemWatcher(ftpPath);

 

            // Add event handlers for file created event

            watcher.Created += new FileSystemEventHandler(OnCreated);

 

            //Begin watching.

     //you need to set this to enable the FileSystemWater to raise events

            watcher.EnableRaisingEvents = true;

 

            // Code to shutdown the console if the user hits 'q'

            Console.WriteLine("Press 'q' to quit the sample.");

            while (Console.Read() != 'q') ;

        }

 

 

Right, so pretty straight-forward so far, a simple console app in which we setup the directory to be watched, implement a new instance of the FileSystemWatcher class then enable it to raise events whenever anything occurs in that directory. Now, the event we’re clearly interested in here is the Created event, which will fire every time a new ftp transfer arrives in our directory. To react to this Created event we wire up an instance of the FileSystemEventHandler to the watcher’s created event and point it to our method OnCreated, which is outlined in Segment 2.

Code Segment 2

        // Define the event handler

        private static void OnCreated(object source, FileSystemEventArgs e)

        {

            // write file name and arrival time out to the console when new files arrive

            StringBuilder sb = new StringBuilder();

 

            sb.Append("File: ");

            sb.Append(e.FullPath);

            sb.Append(" arrived @");

            sb.Append(DateTime.Now.ToShortTimeString());

            sb.Append(". Processing...");

 

            Console.WriteLine(sb.ToString());

 

            //now route the file to where it needs to go.

            ProcessFile(e.FullPath);           

        }

 

Again, fairly straight-forward, the OnCreated method simply reacts to the watcher’s Created event, and allows us to kick off whatever further processing we need to, namely writing out to the console the name and arrival time of the new file, then handing the file off to another method for further processing. Notice the FileSystemEventArgs object in the OnCreated method’s signature – we need to implement this class in order to capture the event, but it also allows us to work with a number of key parameters regarding the watcher.Created event, in particular the e.FullPath property which we’ll use to programmatically work with the newly arrived ftp file. The ProcessFile method is outlined in Segment 3.

Code Segment 3

private static void ProcessFile(String filepath)

    {

        FileInfo file = new FileInfo(filepath);

       

        //Switch to handle different file types

        switch (file.Extension)

        {

            case "txt":                   

                //process text logic here

                break;

 

            case "xml":

                //process xml logic here

                break;

 

            case "csv":

                //process csv logic here

                break;

 

                default:

                    break;

 

            }

 

Ok, so this is the final piece of our solution; we want our console app to be running constantly on our ftp server to watch our ftp dropbox at all times, so we need to be sure it’s as lightweight as possible and doesn’t maintain any internal state whatsoever, otherwise we’re adding extra load to our ftp server, and that’s entirely against what we set out to do in the first place, isn't it?. So let’s not do that.

The ProcessFile method is our routing method to move the file or notify any further services down the line that the file has arrived [an exceptional opportunity to implement the .NET Event Pattern, discussed in my previous post - JT]. We declare a FileInfo class to derive the extension of the file and route the file to a final destination based on file type.  By doing so, our console app never opens the file, never reads it into memory or maintains anything that would drain resources away from our server’s memory or processing pool, so it can run quietly alongside the rest of the server’s workload catching every new file that arrives in our FTP directory, and routing them to their final destination.

Ooo, Aahh.

Now, as simple as this solution is, we need to recognize what this implementation saves us – every time a typical FTP polling process started up, it would need to first gain access to the FTP directory, declare the directory reference in memory, then enumerate all the child directories (even if there weren’t any) followed by enumerating all the files contained in the directory to check if the file the batch process is looking for is there, and because the file transfer cannot be guaranteed to arrive at a specified time, the process would have to execute repeatedly until the file was finally found. Our FileSystemWatcher class, on the other hand simply responds to events that occur within the FTP dropbox, consuming substantially less resources.

So the key for gracefully processing FTP files without having to wait for the entire file to arrive is to process the file asynchronously. Doing so allows the main program to continue receiving file processing requests without having to wait for the code that actually processes the file. .NET provides a number of different avenues to finish that thought with, particularly in .NET 4.0, but that’s a bigger topic I’ll save for a later post. (Oh yes I did)

So to recap, yes we still have to deal with flat file transfers, but at least we can do so in a better manner, can’t we? Oh yes we can.

This coming Tuesday (March 15th) will be a busy one for Definition 6.  Not only do we have World Consumer Rights Day and Andrew Jackson's birthday to celebrate (he'd be 244 if he were still alive), but Definition 6's own ECD, John Harne will be speaking at South by Southwest in Austin.  Additionally, TEDxAtlanta, of which Definition 6 continues to proudly support, will be holding its fifth event.

John's Core Conversation is Right Brain to Left: Art of Persuasive Presentations...
Why so many creative professionals in the interactive design and agency business struggle to sell their concepts, executions and creative work to the business decision makers. What are the potential issues of communicating concepts containing emotion and aesthetic content to analytical thinkers.  How some creative professionals are able to sell just about anything to their clients.

And if you go looking for John in the crowd, this is what he looks like:



TEDxAtlanta's focus this time around will be on Creativity...
Some ideas are too big to capture in language but glitter in our understanding of the world. Creativity is just such an idea.  TEDxAtlanta will explore creativity: not just the beauty, humor and ingenuity, but what's behind it.  We'll experience the wonder but also ask the questions.  Can creativity be measured?  Where does creativity live in our brains?  How can we grow more of it?

Talks will be from:
Sally Hogshead
Elizabeth Turk
Armin Vit
Victoria Rowell
Bonnie Cramond
Margaret Baldwin
Linton Hopkins
Michael Ouweleen
Viktor Venson

And musical performances will include:
Idan Raichel
India.Arie

For the John Harne Experience at Southby, be at the Marriott Courtyard (Brazos 2/3) located at 300 East 4th Street at 5:00PM.

TEDxAtlanta is at capacity... but you can watch the event streaming live by going to tedxatlanta.com. The show starts at noon and goes to about 6:00PM.