Introduction
Over the years of my work in IT, especially in IT sales, I have seen many pilot projects, but most of them ended up in failure with significant time expenditures.
Moreover, when it comes to testing hardware solutions, such as storage systems, there is usually a queue for each demo system that can last nearly a year. Each testing slot in the schedule can lead to a sale or, conversely, derail a sale. It makes no sense to consider a situation where testing does not affect sales, as testing would also be meaningless — it's just a waste of time and an occupation of demo systems.
So, how can we do everything wisely and ensure that it all happens?
Preparation
Pilot Goals
Where does a pilot start? Not with connecting equipment in a rack, not at all. Before any work with the equipment begins, there's paperwork to be done. We start by defining the goals of the pilot.
The goal of the pilot is to eliminate objections from the end customer. No objections — no need for a pilot. Yes, that's exactly how it is.
But what are the main classes of objections that we might encounter?
* We doubt the reliability
* We doubt the performance
* We doubt the scalability
* We doubt compatibility and the ability to work with our systems
* We don’t believe your slides and want to see in practice that your system can actually do all that
* This will all be very complicated; our engineers are already overloaded, and it will be difficult for them
Ultimately, we end up with three main types of pilot testing, and as a specific case of a pilot, proof of concept (PoC):
* Load testing (+ scalability)
* Functional testing
* Fault tolerance testing
In a specific case, depending on the doubts of the particular customer, the pilot may combine different goals, or conversely, have only one of them.
The pilot starts with a document written in clear language explaining why this testing is being conducted. It must also include a set of measurable criteria that unequivocally determines whether the pilot was successful or what specifically was not passed. Measurable criteria can be numerical (such as latency in ms, IOPS) or binary (yes/no). If your pilot includes an immeasurable value as a criterion, there is no point in conducting the pilot; it's merely a tool for manipulation.
Hardware
The pilot can be conducted on demo equipment from the vendor/distributor/partner or on the client's equipment. Strictly speaking, the difference is small; the overall approach remains the same.
The main question regarding the equipment BEFORE the pilot begins is whether the complete set of equipment is present (including switches, data transmission cables, power cables)? Is the equipment ready for testing (correct firmware versions, all under support, all lights green)?
The correct sequence of actions after defining the testing goals is to fully prepare the equipment for testing BEFORE handing it over to the client. Of course, there are loyal clients who are not in a hurry, but this is rather the exception. That is, the full set must be assembled at the partner's site, everything checked and put together. The system must be running, and you must ensure that everything works, the software is deployed without errors, etc. It may seem simple, but 3 out of 4 pilots start with searching for cables or SFP transceivers.
It is important to emphasize that during the verification of the demo system, you must ensure its cleanliness. All data from previous testing must be deleted from the system before handover. It is possible that testing was conducted on real data, and anything could be there, including trade secrets and personal data.
Testing program
Before handing over the equipment to the client, a testing program must be prepared that meets the testing goals. Each test should have a measurable result and clear criteria for success.
The testing program can be prepared by the vendor, partner, client, or jointly—however, it must be done BEFORE the tests begin. Additionally, the client must sign off that they are satisfied with this program.
People
As part of the pilot preparation, it is necessary to agree on the dates for conducting the pilot and ensure the presence of all necessary individuals, as well as their readiness for testing, both from the vendor/partner side and from the client side. Oh, how many pilots have started with the main person in the pilot going on vacation the day after the equipment was installed!
Areas of Responsibility/Access
The pilot program should clearly define and ideally describe the areas of responsibility for all involved parties. If necessary, agree on remote or physical access for vendor/partner engineers to the client's systems and data with the client’s security service.
Pilot
If we have completed all previous points, then the most boring part is the pilot itself. But it should run smoothly. If not, it means that part of the preparation was flawed.
Completion of the Pilot
At the end of the pilot, a document documenting the testing conducted is prepared. Ideally, all tests in the program should have a green checkmark indicating PASSED. A presentation for senior management may be prepared to secure a positive decision regarding the purchase or inclusion in the list of approved systems.
If you do not have a document with a list of completed tests and marks indicating success at the end of the pilot—then the pilot has failed, and it should not have been started at all.
Source: habr.com
