Tender evaluation and linearity

As two of the founding members of Market Dojo are engineers and the other a physicist, we take our maths very seriously.  One of the reasons for developing our auction platform for the public sector during our first grant is that we wanted to demonstrate how easy it was to set up and execute a weighted tender.  The specific area we want to address here is the scoring system.

We have been baffled by the illogical and non linear nature of many of the scoring systems in the market, and were determined to provide something better.

Firstly we would like to describe how we have approached the problem and then look at why some of the other common techniques on the market are challenging to use from the buyer and supplier side and how some are even fundamentally flawed.

Here is a section of our capability; the full version can be seen here.

We think any scoring mechanism should be linear and straightforward.  Thus if you are running a weighted tender you need to be able to clearly communicate the mechanism for converting the various elements to scores and how they are combined.

[You can try out all that you read here by just registering with Market Dojo for free and experimenting with setting up weighted events in the Sandpit.]

Questionnaire scoring:
We have made the set up for a questionnaire very easy, providing many different question types.  On top of this you can create a questionnaire which is scored and also weighted by question, section or both.  The simplest way to show you what this means is an example output.

Here you can see that this questionnaire is scored and also weighted by question and section and the total weighted score is the final result for the questionnaire for this participant.

Price Scoring:
We have two techniques for price scoring on our application.

Pro Rata: The system scores the price based on where it sits between the lowest price and the qualification price. This is a simple linear method of determining differences between prices. The qualification price is set before the event and is the highest bid you are willing to receive. The lowest price is the current lowest bid in the event. [One could look at a further extension to this method by setting the lowest bound but this presents its own challenges].

You might use this method when you are looking at many complex factors and need to have greater control about making the scoring methodology suit your tender. You can judge your price scoring against your highest acceptable price.

Percentage of leading bid: The is a simple and powerful technique.  As an example, if the highest bid in a reverse auction is £100 and the next best bid is £110, then the lowest bid gets 100% and the next highest bid gets 90%, as it is 10% over.  This is also a linear mechanism which does not need a qualification price.  One potential disadvantage is that if a bid is double the lowest bid, the participant will get a score of zero, although that may be a fair outcome.

One benefit of this method is that the non-price elements can be equated directly to price.  For example if the non-price element is worth about 1% of the estimated final price then you can weight the non-price elements to be 1% and the price 99% and the evaluations works out consistently (see the example below).

This method can lead to difficulties in the public sector arising from rogue bids and/ or participants having the same score of zero when they have different bids.  In the private sector though you have more flexibility and can decline participant during an auction.

Combining scores:
The weighting for the price and the questionnaire should be set before the event. The participants bids are ranked on their score, calculated as:
[Questionnaire weighting] * {Questionnaire score} + [Price weighting] * {Price score} = Overall score.

For example:

You can see in the scenario above that the ‘Percentage leading bid’ method works very well.  Here in round 2, one supplier with better payment terms but a worse price has an equal overall score.  The added value of 90 days payment terms compared to 60 days equates very nicely to 1% of the overall score.

Auction:
During the auction you want to easily see the overall score live and to exactly determine how the weighting is effecting the result.

Here you can see the leading supplier based on the overall score:

And you can also drill into the weighted scoring per bid:

The result is an easy to use weighting mechanism that can be used during a live auction with good visibility of all the related scoring..

Issue with current scoring mechanisms on the market:
Whereas any scoring system will have potential flaws, there are some systems on the market which are more flawed than others and some that are simply incorrect.

The multiplier:

This method uses the questionnaire score to create a price multiplier during the auction.  We find it hard to imagine something so incorrect.  There are many issues with this type of mechanism, which we can only think is a result of a software provider not developing an appropriate mechanism but fudging an exchange rate approach to solve the issue,  such as:

•  In a weighted auction, if the participant has a price score of zero, they should still have a questionnaire score.  However, with the multiplier mechanism the overall score will become zero.
•  The methods we have seen for determining this ‘multiplier’ are very much non linear and are impossible for the buyers and suppliers to truly understand how the mechanism would react.
• This method skews non price elements. Let’s say that quality is the non price element used to determine the multiplier.  During the auction the multiplier will create different results based on the price.  Let’s say Supplier A has a price of £10 and Supplier B has a price of £12.  Let’s also say the multiplier is only applied to Supplier A and is 1.1.  In this case the overall score for Supplier A would be £11 to put them in first place.  Alternatively, let’s say Supplier A had a price of £100 and Supplier B with £102.  Here the price is still separated by £2 but with the multiplier applied to Supplier A (£110) they now fall into second place.  In other words the non price element with the multiplier means different things at different prices.

Percentage of leading bid:

For a forward auction, the equation for percentage of leading bid is a simple linear scoring result : ‘Bid divided by Highest bid’.  However, in the public sector we have often seen the inverse of this formula used as the price score calculation in a reverse auction, i.e. Lowest Bid divided by the Bid.  This is mathematically incorrect and actually gives a non linear result, as shown below.  It makes it even harder for the buyer to know how the scoring mechanism would react.

Normalised questionnaire scoring:

We encountered this being used at a council who had applied rigorous OJEU mechanisms to every type of tender, of any value.  In their questionnaire scoring they took the participants’ results and normalised them.  So for example let’s say that the questionnaire is worth 50 points.  Participant A scored 5 out of 50 and Participant B scored 10 out of 50.  After normalisation Participant A gets 25 and Participant B gets 50.  Thus you can see that even if the participant does incredibly badly in the questionnaire, they can still get high points if the other participant has also scored badly.  From the actual points you can imagine how this could drastically change the tender.  In some ways it is saying that there is not room for improvement.  Imagine if the questionnaire was related to quality?  The normalisation could transform a low quality score into a very commendable one, which sadly masks the reality of the situation.

Conclusion:
In summary the mathematics of a weighted tender are actually quite simple and we are proud of the straightforward linear methodology we have developed.  Not only do we believe it is easy to set up but you can also analyse what is going on during an auction.  This is exactly as it should be: fully auditable and compliant.

We are amazed by some of the current mechanisms used in the public sector today. At best they can distort the results in an unpredictable way and at worst are incorrect and non compliant.  Suppliers involved in these tenders might even have the ability to challenge the results which, if you think about it, could create quite a stir.

The biggest disadvantage of these mechanisms is that they are just plain confusing.  Any buyer setting up a tender using these techniques cannot fully grasp how the mechanisms will behave.  And if the buyer will have difficulty, just imagine what the suppliers will think.