Update News for August 2020
Update News for August 2020
Here is a quick run-down on what you will find in this bulletin:
- Back to Court
-
We Got Scraped Again
-
Mission Accomplished – Term4Sale is API
-
API Users Limited To Maximum Quotes
-
New Updating Service For Engine Users
-
Web Quote Option Still Best For Most Subscribers
-
So Why The API?
-
The Other Reasons For The Cap on API
-
Canadian API
- CQS.EXE is Our Next Project
- Our Current Programming Plans for 2020
These topics will be dealt with in more detail throughout this bulletin.
For those who have followed the saga of our legal fight with the 4 individuals who we sued for copyright infringement, the matter is now being scheduled for a new, supplementary trial following our win in appellate court. The court of appeal has tasked the lower court with correcting the errors in the previous legal decision.
In order to address that the parties have agreed that instead of having a completely new trial we will have a supplemental trial to address the additional concerns raised by the court of appeal.
The challenge will be that the new trial will be largely on-line, given the covid issues in Florida where the case is being heard. Whether than means the public can have access to the trial, in the same way that the public can attend a proceeding normally, I don’t know. But if access is available I will let you know. It might be worth watching for the more curious among you, especially if you are stuck in the office.
You know that you have good stuff when people keep trying to steal it. During mid-July we had a new robot trying to steal quotes from our term4sale.com website. However, unlike the previous robots that were trying to talk directly to our internet engine, and obtain quotes without using the client entry page, this time the robot was sending values to our client entry page then producing a quote in order to scrape that.
Two things kept this from going very far.
First, that method of talking to our internet quoting system slows down the robot. In the previous scrapes, where quote request was being transmitted directly to Compulife’s internet engine, the pirates were getting about 8 or 9 multi-company quotes per second. This new scraping effort slowed it down to about 10 quotes per minute. The reason for this is that each layer of software, between the user (in this case robot) entering in client data, and the production of a quote, will slow down the process. While the internet engine still goes as fast as it ever did, no longer having direct access to the engine means pirates have to go through the same steps humans do when using the quoting system normally. And while to a human there is hardly any discernible difference in performance, it’s a definite slow down to a robot that is spitting out requests like a machine gun.
NOTE: There is now a new option that we sell to life insurance companies and actuarial firms called the “Compulife Batch Analyzer“. Qualifying subscribers who are purchasing that option are able to generate something in the order of 100 quotes per second. Those results are then dumped to a spreadsheet where the subscriber can do further analysis. The reason that is so much faster is because there is a much closer connection between the sending and receiving functions. When the program can process all that internally, and there is no internet transmission involved, the program can run at full speed and Compulife’s Batch Analyzer is super fast. It’s a real testimony to the quality of our programmer’s work that our software can produce quotes at these speeds.
The second reason the scrapes did not get far was that the new reporting systems connected to our API function alerted us to the unusual volume of activity and we were able to block the IP. Once we realized that someone wrote a robot to come in the front door, we analyzed it and moved forward to create an automated defense mechanism. Once the system recognizes inordinate numbers of requests coming in from a particular IP address, regardless of which customer’s API account or accounts are being hit, the software now cuts off quote production at a certain “cap” for the balance of the day. Even if hackers try to use this newest mechanism to scrape, the problem they face is that we can stop it no matter which of our customers or group of customers are being raided.
It remains a real shame that we have to spend so much time programming defense mechanisms, rather than turning all of our attention at creating a better product for our subscribers.
There is the old saying. “All good things come to those who wait.” But Chris and Jeremiah have not been waiting, they have been systematically building and improving the API to the point where we couldn’t break anything anymore. In mid-June we flipped the switch and the API went live at term4sale.com. And for the last two weeks it has been active, we have not heard a peep from anyone about issues or problems. Great job guys!
You can tell that the new API is running because the site is now producing 75 product results instead of the previous 25. In the past we were producing 50 results but then after we discovered our site and our engines were being targeted and scraped by the NAAIP operation, who was running a scraping robot at term4sale and other sites that were using our internet quoting engine, we cut back the number of results to 25.
Compulife’s Internet Engine remains the most powerful quoting tool that we provide. If you read the previous story, you should realize that having that on your own server allows faster performance and so customers with large volumes of quotes to produce will find the best performance from the quote engine residing on their own server. You do not have to communicate requests for a quote to a third party server as is the case with the API option.
The problem for some existing engine customers is that there is a maintenance issue that goes with having the internet quoting software on their own server. The problem is that when Compulife updates rates, through the normal update process used to update your PC software, the Internet Engine user must obtain the rate update using the Windows PC software, then upload the PC data files to their server, so that their internet engine now has the same updated files.
Some subscribers find this process a pain. It is also a problem if they have a turnover in the people they use to do this, whether they have delegated that job to members of their own staff or to their website developers.
The big advantage of the new API is that while all the user files that control request and display of quotes now reside on the customers server (as they do with the internet engine), the raw quote information is coming from Compulife’s server. And because Compulife keeps our servers (we have more than one) updated with rate changes as we publish updates to the PC software, there is no lag time in getting updates to our quoting service on-line.
One of the limits that we have decided to place on our API quoting function is that the API option is ONLY available to those who do 30,000 quotes or less per month. Now folks, that’s a lot of quotes per month! How do we know that? We based that upon our own term4sale website that might do a maximum of 1,000 quotes per day during weekdays (all time peak) but typically will slow down to half that on weekend days. Our quote numbers did run up during the scrape that we detected after the implementation of the new API function, but it is hard to imagine term4sale doing more than 30,000 per month.
So for customers doing more than that 30,000 per month, they will need to continue to use the Internet Engine on their own server.
To address the updating issue, Compulife is now offering a new service to Internet Engine Users where you can hire Compulife to update your web server with the latest updates. The cost of this new service is $360 per year which averages to $30 per month. You will need to have your web developer create a password access to your server, and to the folder that contains your Compulife data files (we don’t want any further access than that). The access must allow FTP communication with that folder; once again, only for that folder.
On Compulife’s end we will install a copy of your edition of Compulife and keep it updated each time that we do a monthly or midmonth update (just as we do for our own servers that offer Compulife web quoting services). As part of our update routine, once a monthly or midmonth update is introduced, we use the PC copy of the software to process the update file and then using our FTP software, copy the changed data files to our web server. Note, different servers use different serialized copies and so each server gets it’s respective files.
Anyway, we are now prepared to offer that same service to those who would like us to do it for them. Once again, the cost is $360 per year, which we think is quite reasonable and which should alleviate the biggest complaint that we have with engine customers.
And for those who think that’s too much money, and who are doing less than 30,000 quotes per month, the API service will be your best course of action. You can eliminate the need to do updates, and save money on the costs of quoting versus the Internet Engine. And for those customers who are doing their own updates now, and have no problem, nothing changes.
When we announced the new API, we did it by email to all existing Internet Engine and Web Quote subscribers. The roll out of the API option came with a special deal for the first 12 subscribers that sign up and pay their $300. We wanted those who are using those existing web services the option of having first kick at the can.
Now that those subscribers have had that kick at the can, the offer is now open to all existing subscribers of Compulife and to new customers until the cap of the first 12 to purchase has been reached. Here is the email that Jeremiah sent out in July:
Some of our web quote customers emailed us to ask whether they were required to update to the new service and if the web quote option was being terminated. NO, the existing web quote option remains. Some asked if the API was something they should upgrade to, and my response was this question, “Are you happy with the way the web quote option is working for you now?” To those who said they were happy, we then replied there is NO need to upgrade to the API, keep what you have.
The beauty of the web quote option is that it allows you to easily add a web quoting program to your website, and putting it on a page can take as much as 5 minutes if your page developer is slow. All that has to be done is that an “iframe” is made on the page (a box, like you would create for a photo) and in that iframe you place the link we give you for your quoter. The biggest part of the job is sizing the box so that it looks good. The biggest chore after that, is creating the content which will surround your quote option on the page.
And the web quote option is cheap. You only pay $96 per year, unless you start loading it up with other options that you can see here:
Good question, and the answer is market demand. We increasingly hear from new customers the question, “Does Compulife have an API?” And what they really mean (they are asking the question their web developers asked them) is does Compulife have an API which delivers information using JSON formatting standard. The answer to that question is now, “Yes”.
Never mind that the prospect could have just bought the web quote option, made a frame and stuck it on their website for less money. The problem is that your web developer can’t make much money doing a 5 minute job, and so they hustle you into thinking you need the API quote option to do a bunch of fancy things that you may or may not actually need. So now we have a product option that lets you buy, or lets your developer sell you, something that can let you add all the bells and whistles that you want, it’s JSON compatible.
And the API is better for Compulife in that we now have a database gateway that monitors all activity going to our Internet Engine. Yes folks, at the heart of the API is the same old, good old, internet engine that is now surrounded by a database that acts as gatekeeper and monitor.
There are two more reasons we are capping quote volumes on the API.
First, we don’t want one super high volume customer dragging down performance for all our other customers, and we don’t want all our other customers dragging down performance for that high volume customer. Periodically something will happen on the Internet that creates huge volumes of traffic, and when that happens you will hear that a website has “crashed”; think “Obamacare” roll out.
Volume can become an issue. Of course internet transmission speeds have been going up, and you can now stream an entire 4K movie through the web which is a ginormous amount of data. But from time to time you’ll be watching your Netflix or Amazon show and you will notice that all of a sudden your “hi-resolution” image degrades and starts to look like what you remember your old 28″ TV looking like. What’s going on? The volume of use between you and the server that is providing the movie has been compromised, and the service has been downgraded to reduce the volume of data being transmitted.
And Compulife doesn’t want a high volume user to cause that to happen with our quote transmission. If you exceed 30,000 quotes per month, you need to be running quotes off your own server. And one customer doing 300,000 quotes per month might not affect us, but we aren’t talking about one customer, we are talking about the potential of hundreds of customers.
The other reason for the CAP is that we don’t want the larger volume user dependent upon our server for “up time”. No matter who you get your internet services from, ALL internet providers are going to experience down time.
For example, Here at my office at home I have TWO internet services and TWO routers for those service. One is a DSL, the other is a cable TV router. Why two services? Because I need to stay up all the time and inevitably one of those service will fail. Sometimes the problem is fixed in an hour, sometimes it takes 3 days. I can’t tolerate that kind of “wait time” for repair. So what I do when it happens is I unplug my computer from the one router, and plug it into the other router. Problem solved until the failed service is fixed. And because we have more than one computer at the house, and because I keep different computer on different routers, I know when there is a failure.
NOTE: This is just an example of a failure that can occur on the web. Compulife has NO servers that are connected to the web by DSL or cable service. The servers we used are rented from large IP’s who connect to the web with fibre optic. It is worth noting that our updating service, and our quoting service options, all run from multiple IP’s that give us a broad, manually controlled redundancy. Why? Because I don’t trust computers and I don’t trust the Internet.
Anyway, we do not want a failure of one our servers to cause an interruption in our service to a high volume customer, such that the customer is now UNHAPPY with Compulife when the problem is outside our control. When you license the Internet Engine option, you are placing the quoting software on YOUR server; Compulife is not in the loop for the hardware that is running your quotes. If you then have a web based interruption in service, that’s NOT because of Compulife, that has to do with your Internet Provider. Compulife is not the problem.
While building the U.S. version of our API, which is implemented at term4sale.com, Chris has also been building the API for Canadian customers. Chris and Jeremiah determined that running it all from a single database would not be that difficult, and that it would be much, much easier to administer.
After implementing the new API for term4sale.com, Chris and Jeremiah rolled out the new API for term4sale.ca.
We are delaying the email with the introductory offer to Canadian subscribers until sometime later in August. We want to see what issues we might run into as U.S. subscribers work through implementation of the API on their own site.
With the API in the rear view mirror, we are now re-focusing our attention on the next area of work which is to upgrade our PC software to a new data structure and a new version of our program to go with it.
In that regard the following is the current order for new work that we will be doing in 2020. Please note that I have switched what was number 2 with number 1.
-
-
- Introduction of New PC Version:
-
CQS.EXE
-
-
- Overhaul Of Current Product Data Files
-
-
- Introduction of Compulife Basic Plus (with Pick 12)
In giving this further thought, we decided to move up the creation of the new Windows program to the top of the list. That new program will be called CQS.EXE and will eventually replace the current program GOWIN.EXE.
One of the things we hear from new prospective customers is that they think the existing program looks old and outdated. Yep, it looks old, but is hardly outdated. Do you own a handsaw? Yep. Is it old and outdated? Yes and no. Are there newer saws that work better? Yes. But you still have that old handsaw, why? Because it’s dependable and nothing changes the fact that it will work regardless of the circumstances. You don’t have to worry if you forgot to charge the battery or whether what you are cutting is too far from an electric box. You just pick it up and use it.
And for many of our existing subscribers GOWIN.EXE is an old dependable tool. It works, and it gets the job done.
And that’s the logic I use when I think of the DOS interface on my Windows 10 computer. Why do you think that Microsoft still offers that old clunky DOS interface as an option? Simple, there is NOT a day that goes by where I do not open a DOS windows and do something on my computer using the DOS interface. And there are some days when that’s just about all I do. And there are some things you can do with the DOS interface, that you can’t do with Windows and I am not alone out there. But I digress…
The main reason we are changing the Windows program is because we want our Windows PC software, and our web based Compulife Basic to eventually look and work as much alike as possible. And because there are some things a browser will not let you do, which our Windows program does let you do, it means approaching some of the functions in our software in a different way. Our objective in changing the software is NOT to remove a function or the way you use it, but to add new ways to do functions such that those same functions can be done the same way in the web based version of Compulife.
Our ultimate goal in doing this is that eventually Compulife Basic will be able to do all the things that our Windows program does, so that the web based user can have all the same capabilities of the PC software. And by the way, at some point, when we enhance Compulife Basic, there will be a new version called Basic Plus that will sell for more money. Note that the PC subscriber, who gets Basic for free with their subscription, will also get Basic Plus for free with their subscription. But Basic Plus is down the road.
Before we get to that, we want a PC version of our software that will replicate what we know we can do on the web, the way it will have to be done on the web.
We originally planned to roll out the new PC version with the new data structure, but now will roll it out earlier. We want to have customers start working with it before the unseen part of our software changes. Rather than deal with two things at once, this will let us deal with one thing first, and the second thing after.
It will also give a period of time for customers to use old or new versions of the software, as they prefer, for a longer period of time. This will make the learning curve simpler. Once the CQS.EXE is hooked up to the new data, we will begin the process or ceasing support for the older version and you will need to use CQS.EXE instead of GOWIN.EXE. Again, this strategy give your more time to make that transition.
More news on this to follow.
The following is the current order for new work that we will be doing in 2020:
-
-
- Introduction of New PC Version: CQS.EXE
-
-
-
- Overhaul Of Current Product Data Files
-
-
- Introduction of Compulife Basic Plus (with Pick 12)
Anyone with questions about any of these upcoming projects can call Bob Barney to discuss:
Please don’t email me essay questions, just call. If I’m not in, email me your phone number, I’ll call you.
These planned objectives will easily consume our programming time during 2020. The good news is that once the product data files have been converted, and we have introduced the new CQS.EXE, and upgraded our internet engine to use the new data files, Compulife will be turning it’s full attention to our web based, Compulife Basic software. The long term goal is to have a web based product that does everything our PC based software does.
Related Posts