Aug. 20, 2009 - Completing RETS: The Survey

It's difficult to write about opportunities for the future of the RETS standard without some folks thinking I'm bashing the past. To be clear, I honor and respect the past - but also welcome the future. Discussing the possible future of the RETS standard and engaging stakeholders can only make that future better. So, I put up a short survey for MLS executives to let their voice be heard.

In my previous post, "Completing RETS" (http://www.realtown.com/mattcohen/blog/completing-rets) I pointed out that there are some types of data that it would be ideal to represent using RETS to achieve the business objective of making it easier to move from MLS system to MLS system or to move data from MLS systems to other systems where the data could be used. When asked how important each data type was, here's what the MLS executives said:

Documenting and being able to transfer MLS business rules via RETS was the most strongly desired of the data types I queried on.  Every MLS executive has gone through some pain moving these rules from system to system. As I had previously commented on my blog, how to work with or incorporate some type of business rule markup language is an area ripe for discussion. Being able to move prospects and saved searches from system to system was also strongly desired - end-users not being able to move this data from system to system is a part of what makes MLS system transitions more difficult and time consuming for MLS subscribers. Thinking beyond the MLS, being able to move such data from broker and agent websites into the MLS - or from an MLS public website to a broker/agent website once the consumer has selected a professional to work with - would be wonderfully convenient. Customized search screens in the MLS are not that commonly used, so that didn't rate as highly. David Harris from eNeighborhoods had suggested that Open Houses was an area that needed to be better represented in RETS, and while the MLS executive segment didn't see it as that important, that non-alignment may point to the need for even wider discussion and measurement-taking, involving other stakeholders.

A supermajority of survey respondents wanted to see data types they ranked as 'Very important' or 'Important' to be added to RETS over the next year. I'm not sure that's possible given the current resources of the RETS community. To quote Mike DelGaudio, “Finishing out these remaining schemas …  is can of worms that many implementers won't be prepared to execute quickly, affordably, or safely.”  And, as one respondent said “I rated all the elements as very important and want them all this year. Will that happen? Heck no, and I know it. … RESO has [taken] years to approve what we have now. … After years of trying we don’t have common set of data definitions. If RESO can’t get that done, how on earth do you expect them to tackle some of the "people" fields (to use DelGaudio’s term).”  I agree – this is a challenging endeavor! However, if the interest is there and scope and time are truly fixed then MLS executives could discuss the third part of the project equation, additional resources, with NAR. This is all part of creating a project plan for RETS - a roadmap that we can measure progress against.

That leads to an interesting subject – the RETS roadmap. 26% of the MLS executives surveyed said they were aware of the “long term roadmap for RETS”. That’s really fascinating, because there’s no such thing. There was one back before 2007 but once we left the road heading to RETS 2.0, the roadmap was stuffed into the virtual glove box (just try finding the roadmap on rets.org) and the focus of the effort turned to small-scale changes and, based on the immediate needs caused by the NAR policy, compliance. I confirmed the lack of a long term roadmap with some RESO board members.

On the subject of MLS executive stakeholder involvement in the standard, the majority of MLS executives have not been to a RETS meeting and only 34% of those surveyed said they understand the process for making changes to RETS. The good news – of those five executives that said they had tried to get a change to RETS made, four said that they were successful. The process can work if MLS executives and other stakeholders want to get involved!

While surveying MLS executives on the future of RETS, I did ask some questions about the present that might interest the reader:

58% of respondents said that RETS has made accessing data easier for subscribers and only 16% said it made it more difficult.

• I would say that once they understand the process they like it better, but it is getting them to move to the new process that is more difficult.
• Most Brokerages are required to find vendors who understand RETS. Small vendors, in most cases, don’t have a clue.
• The vendors all seem to have different problems, they blame the MLS programmers. It seems that the different RETS clients cause different problems for the vendor which in turn causes problems for the MLS.
• Too many web masters don’t understand it.
• When we switched from one platform to another (through the same vendor) the agents still had to reenter all their saved contacts, etc. The possibility that this could be different would be a huge improvement.

• More online resources to help third parties get started with RETS.
• MLS administrators need more training as the subscribers seem to be learning RETS as well and a difficult time implementing it.
• Develop a standard RETS Client

As one respondent said, “[RETS is] easier for staff to provide MLS® because they only have to set permissions and criteria once. It’s harder for staff because now have they must liaise with 3rd Party Providers educating them as to the different "flavour" and/or implementation of RETS.”

When asked what RESO could do to make it easier, respondents had a number of comments – but most of them seemed more focused on their technology vendors rather than the standards effort itself:

• Get the vendors to understand that RETS is RETS and the MLS does not have to customize for each vendors RETS client.
• Allow more filters; i.e. a way to "throttle" the data.
• Vendors using RETS need to change their model of operation. Instead of open access for each subscriber, vendor should have an authenticate standard and then all queries are done by the vendors account, instead vendors say turn RETS on for each subscriber creating security issues.

Of those surveyed, a 53% majority still serve data primarily using FTP or some other method – hopefully this will improve over time.

There has been a lot of great discussion lately about charging for RETS, reflecting my 2003 paper on the subject (for which I took so much heat). It’s nice to be validated six years later!

One respondent clarified: “The charge is only for third party vendors who sell product to members. The real issue MLS’s need help with is defining the categories of data subscribers. We don’t charge members for RETS feeds, other than a criteria change fee.”

Let’s end this post on a really positive note. I asked MLSs what benefit they’ve gotten from RETS. There were some negatives to start with, but most of the comments were quite positive.

• I think the true test will be what impact it will have on our servers and bandwidth issues. Also updating to more current versions seems to be an issue.
• At this point none.
• Have not seen any benefits to our MLS. The programmers seem to spend more time on RETS than they do programming the MLS System for the members.
• RETS hasn’t added anything except another way to transfer data. We did fine with and ftp feed and framing. It might have made a difference to some IDX vendors, but was no big deal to MLS.
• If you ask an agent how they get data to their site, they couldn’t tell you. They don’t care as long as it gets there.
• We use ftp to send data to most third-party vendors. For the most part RETS access is used to provide members with another means of data/photo access other than our MLS vendor’s standard front end. One reason we use ftp to service third party vendors is that with RETS my staff frequently has to spend time helping the vendors understand our data structures and resolving issues with their RETS client and query structures/syntax.
• To up a RETS data feed is much, much too confusing. IDX … has an extremely straight forward up process - terminology and the layers to things up in RETS makes no sense to non-programmers

The more positive responses:

• For vendors who are familiar with RETS, they are up & running with their application within days, instead of weeks or months.
• CARETS
• More flexibility with the data, more timely updates and easier marketability for our products.
• Got rid of the ftp servers and closed the security loophole that ftp created.
• Created data standards for accommodating data from different marketplaces; created modules for our MLS application which can be unbundled and plugged into other RETS utilities
• Simplified support to vendors, stability in providing data, not needing to provide various versions as in FTP files, easier for image access (we prefer that vendors pull via http (URL supplied by RETS) versus accessing photo via RETS - saves time, bandwidth etc., able to offer real time data
• Expedited, more secure, and manageable means of data transport.
• Better support and access to real time data
• Better, more accurate and standardized feeds available to our members.
• The move to RETS has made the prospect of data sharing with neighboring systems more straight forward. The neighbor can pull down the data on their own with limited use of our programming staff.
• We have been easily able to partner up with third party aggregators.
• We’ve had a few venders that will only do RETS, so this has opened a door of opportunity for them to present their product to our members.
• List Exporter (self hosted) uses RETS to provide filtered IDX files which requires very little policing of IDX sites.
• Depending on the service - brokers might pay extra.
• For the vendors serving our subscribers it is very valuable.
• Much easier to implement and provide from our side. Non-dues revenue - increased choice for members dealing with 3rd Party Suppliers - better controls for access to MLS® data
• We use it internally to calculate some statistics and to feed a MLS Data Checking tool. It feeds our public website and a good portion of our broker IDX feeds. I agree with the responses to the blog, facilitating the transport of saved searches and such is a very big deal.
• It’s important for the future of real estate to have a standard that is going to put everyone one on the same page. RETS has opened a lot of doors as far as data is concerned for us and all MLS’s that are utilizing it but it’s a standard so I believe it will change as needs and demands evolve!

RETS has come a long way over the past ten years on the backs of hard-working volunteers – but there is room for RETS to mature further. Involving stakeholders and looking at business objectives with fresh eyes will be important as we move forward. Putting more resources behind the effort and continuing to formalize the requirements gathering process, the road map, and the project planning and communications side of the RETS effort will be crucial, if one wants to see more than small incremental changes to the standard made over a long period of time. Working to restructure the effort and providing the resources needed to do so will require resources for RESO that MLS and Association executives will need to advocate for, if they have an interest in RETS moving forward in ways outlined in this article.

The Fall RESO/RETS Conference is September 23rd - 25th, 2009 in Chicago, IL - more information is available here: http://www.rets.org/meeting

Aug. 11, 2009 - Why Do We Need Another RETS Vendor?

The rise of third-party RETS servers and the emergence of a competitive marketplace

Many MLSs are licensing RETS servers in addition to the one provided by their MLS vendor. Why are we seeing this trend? Reasons include the desire for:

• Control over speed and reliability of the RETS server
• More sophisticated controls over the data exposed and formats it is exposed in
• Enhanced monitoring and reporting
• Additional security controls
• Use of RETS beyond the listings – accommodating other data types 
• Customization beyond what the MLS vendors are providing
• Not having to change RETS servers and support data recipients when changing MLS vendors

This is not to imply that the MLS vendor RETS servers are "bad" – they sometimes just serve different needs and have different capabilities than third-party RETS servers that one might have to pay for. To be frank, most MLS organizations are not paying the MLS vendors sufficiently for them to add all of the types of features one sees provided by third parties.

The primary players in this space are Bridge Interactive Group (RETS IQ) and Threewide (ListHub and ListExporter). Recently Northern Ohio Regional MLS (NORMLS) has also entered the market. Following are profiles of each company and their products.

Bridge Interactive Group was established in 2004. Bridge’s core goal is providing RETS management and enhancement tools, customized RETS solutions and facilitating data sharing initiatives.  Bridge employees actively participate in many RETS workgroups and they hold a Chairperson seat as well as a seat on the Board of Directors for the Real Estate Standards Organization. 

The RETS IQ Suite of products is designed to put control of data management into the hands of the customer and includes the following components:

• RETS Contact™ is an MLS level RETS filtering, monitoring and management tool that adds further fine-grained control and usage transparency to your current MLS RETS system.
• RETS IQ Cheque™ is an add-on Contact application built to administer billing and account management services for MLS RETS Data Recipients.
• RETS IQ Accelerator™ is a middleware product intended to add reliability, scalability and performance to RETS servers by providing an elastic cloud network of RETS server caches.
• RETS IQ Pulse™ is a RETS server tool for MLSs and Server Providers to monitor their RETS servers. With Pulse, notifications are immediately sent the minute problems with the RETS server are discovered.
• RETS IQ Compose™ is a rich web application for adding and editing listings via RETS 1.x. Compose is almost completely driven by RETS metadata including field layout and validation.

Bridge Interactive Group provides a lot of services around these products, including hosting, setup, data mapping and support.

Some of the key differentiators of the RETS-IQ suite include the ability to create and use data “views” (I’m not going to get technical in this article – but it’s important), the ease of identifying excessive use and getting paid exactly what one should be for that use, the architecture needed for enhanced speed and reliability, and the tools one needs to monitor and assert control over RETS service levels.

Recently Bridge Interactive Group launched RETS IQ Rosetta - a free cooperative RETS Metadata mapping service that Bridge Interactive Group is offering to the RETS community and in particular to RETS client vendors and developers. Its goal is to lower the technical and domain knowledge entry level for new developers in the RETS world, promoting innovation and development of new RETS applications.

The Bridge Interactive Group’s RETS IQ Suite of products now manages MLS listing content for more than 150,000 real estate professionals nationwide via 11 implementations.

Threewide Corporation was founded in 1999 and their ultimate goal is to make possible a single point of data entry that powers the many diverse software systems used by real estate brokers, agents, and administrators in their daily business.

Threewide's primary product is ListHub, which provides a mechanism for easy listing syndication. A broker/agent interface allows for opt-out from listing syndication to specific sites, which Threewide has arranged with over thirty of the most popular national real estate sites and a number of regional portals. ListHub syndicates not only the listing information, but unlimited photos, open house and virtual tour information. MLSs can select where links from the syndicated listings go – choosing from a default redirection to MLS consumer facing site property pages, ListHub property pages, or other property pages of their choice. ListHub offers advanced reporting on the exposure given to the listings - not on the sites they have syndicated to - but via default landing pages for each listing that ListHub provides, assuming the syndication sites link back to the Threewide site. Threewide also tracks and provides reports on the leads sent from their landing pages. Reports are available at MLS, franchise, brokerage, office and agent levels, as well as a branded report for the consumer for their individual listing.

Threewide’s ListExporter enables the MLS staff to selectively package listing data and images and send them to virtually any location in a variety of custom and industry-standard formats. With ListExporter, an MLS can control where the data is being sent, either on demand or through a set schedule. ListSecure works in tandem with ListExporter to minimize the unauthorized use of listing data and images through industry-proven data tags, image watermarks and secure file pickup. Extensive audit trails show when data and images are retrieved and by whom to easily identify the misuse of data.

Some of the key differentiators of the Threewide products are having the most “out of the box” syndication feeds (31 national, plus 16 regional) and the data tagging or “watermarking”.

Threewide’s ListHub product is in use by 190 MLSs.

NORMLS, Ohio’s largest multiple listing service, formed a strategic partnership with Ronin Technologies (http://www.ronintech.org/) to launch a new product called RETS Genie®. NORMLS serves 750 member real estate companies and their 8000+ agents. Ronin Technologies has been a leading provider of RETS solutions for over ten years, was heavily involved in writing the RETS 2.0 specification, and has chaired the RETS compliance workgroup since its inception in 2003.

RETS Genie® was developed to enable the MLS and its members to create data packages for advertising, applications, and custom requests. RETS Genie®is available in two versions, Standard and Deluxe. The Standard Edition enables MLSs to aggregate, integrate and organize any data and photos available through RETS. An MLS can configure reusable IDX filters and define delimited and ‘zip’ compressed file exports of this data for members. Data exports can be run on demand or scheduled to run, then automatically sent via FTP or HTTP to locations configured by the MLS. Users can choose to send either “full” (all the allowed data) or “incremental” (only new or changed data and images) updates to data recipients.   The Deluxe Edition additionally allows MLSs to configure custom database columns and tables for RETS import. Data can be syndicated to Trulia, Yahoo, Zillow, Google in their preferred format, or to any other site using the RETS Syndication schema. Since most syndicators require a unique URL for listings, syndication support includes a unique landing page for property details. The Deluxe Edition also offers the capability to deliver data via email or to a file system on a network. 

RETS Genie®offers add-ons to the Deluxe Edition. The Data Source Module enables integration of other data sources (i.e. member management systems, public records, or tax data) in addition to RETS. The MultiFormat Module enables exports to Excel, HTML, PDF and RSS formats, as well as custom XML formats. The Rules Module facilitates the creation of business rule filters to support VOWS, reciprocity, and third party agreements.

NORMLS offers hosting for RETS Genie®, and partners with Ronin to provide installation, data mapping, training and support.

Some of the key differentiators of NORMLS’ RETS Genie® product are that it easily  incorporates data from non-RETS sources and supports a wide variety of custom output formats including Excel, PDF, HTML, and HTTP and email delivery options.

I am pleased to see a competitive marketplace emerging for RETS products, and has written this short article to recognize a few of the many creative and talented industry colleagues that are blazing this trail for our industry.

The RETS standard has come a long way over the past ten years, but the promise of RETS will only be realized through product development that leverages the standard. In much the same way that concepts of hypertext first demonstrated in the 1960s did not achieve their fruition until the 1990s when the Mosaic web browser application blazed the trail for the World Wide Web, it will take products – killer apps – to really bring RETS into its own golden age and create opportunities to enhance the industry with new kinds of information-hungry applications that are only now being imagined.

Aug. 7, 2009 - Completing RETS

A standard, like software, is never complete - it just goes on to the next version. But, the RETS standard has been going on for ten years - I think it's time to take stock about how far it has come and talk about where it could or should go.

If you're an MLS executive or an agent, why should YOU care about this RETS stuff moving forward quickly? Doesn't it already facilitate your data feeds, syndications and MLS collaborations? Think about this - recently I was negotiating an MLS contract and wanted to know what data, created by my client and their subscribers could be moved from the old system to the new one:

• Listings? Of course.
• Roster? Of course.
• Contacts? That might be possible with custom work, not using RETS.
• Prospects? No.
• Saved searches? No.
• Custom columnar reports? No.
• Custom search screens? No.

More than half of the items that we wanted to move from system to system using RETS couldn't be done. Each one we couldn't move would mean pain (time/effort) for the MLS subscribers as they had to re-create the data in the new system manually. This isn't just relevant to moving from one system to another - it's relevant to offering a parallel system - or multiple MLS front-ends - or any number of potential MLS add-on products. And I could make similar points about the difficulty of moving between other real estate systems, such as association management and transaction management systems.

This difficulty in moving systems or adding systems (some would call this the "friction  of the transaction") does not encourage the most competitive marketplace for real estate technology - and this is bad for my MLS and Association executive clients and bad for their subscribers and members. Face it - most of the people driving the data standards are more established vendors - and they do not have incentive to improve standards to make it easier to move to another system. So, if you want the standards completed to the point where you can move *all* of your data from system to system, increasing real estate application innovation, increasing competition in the technology marketplace and making transitions between systems easier for your members - YOU have got to work with your peers to drive the standard forward.

If you want to contact the Chair of the RESO Outreach and Education committee, email Kristen Carr (kcarr@big-llc.com).

The Fall RESO/RETS Conference is September 23rd - 25th, 2009 in Chicago, IL - more information is available here: http://www.rets.org/meeting